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EXECUTIVE  SUMMARY 

Background:  The  Health  Care  Financing  Administration  contracts  with  peer  review 
organizations  (PROs)  to  obtain  physician  review  of  medical  records.  While  the  results  of 
individual  case  review  by  one  or  two  physicians  have  been  recognized  to  be  unreliable,  and 
HCFA  no  longer  contracts  with  PROs  to  act  on  results  of  individual  reviews,  results  of 
review  of  groups  of  patients  treated  by  a  given  physician  or  provider  are  used  by  HCFA  to 
help  physicians  and  providers  identify  quality  improvement  opportunities. 

To  make  the  review  process  more  efficient,  it  has  seemed  desirable  to  use  a 
screening  strategy  to  decide  which  records  should  be  referred  to  a  physician,  thus  utilizing 
more  expensive  physician  time  most  efficiently.  Such  screening  systems  have  received 
little  evaluation  and  rarely  have  alternatives  been  directly  compared.  HCFA  is  developing 
the  Uniform  Clinical  Data  Set;  one  function  of  this  data  set  was  to  provide  automated 
algorithms  to  screen  cases  for  whether  they  should  be  referred  for  review. 

Aims:  The  purpose  of  this  study  was  to  measure  the  sensitivity,  specificity  and  predictive 
value  of  three  different  medical  -record  based  screening  systems  for  quality  of  care 
problems  experienced  by  patients  with  coronary  artery  disease  hospitalized  for  myocardial 
infarction  (MI),  percutaneous  transluminal  coronary  angioplasty  (PTCA)  or  coronary 
artery  bypass  graft  (CABG)  surgery:  The  three  screening  systems  were: 

1)  an  early  prototype  of  the  Uniform  Clinical  Data  Set  (UCDS)  that  was  being 
pilot  tested  by  several  peer  review  organizations  at  the  time  the  study  was  undertaken; 

2)  Johns  Hopkins  hospital-wide  (JHH)  screens,  including  HCFA  generic  screens 
that  were  used  by  many  peer  review  oganizations 

3)  the  Harvard  Medical  Practice  Study  (HMPS)  screening  criteria, 

The  study  originally  proposed  to  measure  the  screening  systems  against  two  different 
reference  standards  ,  blinded  to  the  screening  results:  1)  agreement  by  at  least  two 
reviewing  cardiologists  that  there  was  a  quality  of  care  problem,  2)  the  determination  that 
an  adverse  event  was  due  to  negligence,  as  defined  in  the  Harvard  Medical  Practice  Study. 

Methods: 

A  stratified  sample  of  460  medical  records  was  drawn  in  three  strata:  myocardial 
infarction,  coronary  angioplasty,  and  coronary  artery  bypass  surgery,  from  admissions  in 
the  period  7/1/89-6/30/91  at  Johns  Hopkins  Hospital.  Within  each  cohort,  the  sample  also 
was  stratified  by  whether  or  not  records  had  passed  Johns  Hopkins  Hospital's  hospital- 
wide  quality  screens.  Records  were  screened  using  all  three  screening  systems  by  trained 
quality  assurance  nurses.  All  records  were  also  reviewed  by  two  to  three  cardiologists 
using  structured  implicit  review  instruments  to  determine  the  reference  standards.  All 
records  were  also  coded  for  APACHE-III  severity  of  illness  scores  and  the  Charlson 
comorbidity  index.  ROC-like  curves  were  obtained  using  different  subsets  of  screens  and 
the  area  under  the  curve  compared  for  these  different  subsets.  Nine  records  were 
unavailable  for  review  and  twenty-eight  were  reviewed  by  only  one  physician  and  were 
excluded  from  the  study  results  that  use  two  physician  reviews. 


Results: 

Agreement  among  physicians  about  whether  there  was  a  quality  problem  or  not  was  76% 
(kappa  0.37).  Both  physicians  agreed  that  there  was  a  quality  problem  in  17.8%  of  cases, 
pooling  all  three  cohorts,  and  adjusting  for  the  stratification  procedures.  In  all  three 
screening  systems  a  small  number  of  screens  detected  most  of  the  quality  problems. 
Overall  agreement  in  the  nurse  screening  result  about  whether  to  refer  a  record  for 
physician  review  was  98%  for  UCDS,  77%  for  JHH  ,and  85%  for  HMPS  .  The  systems 
performed  as  follows  in  detecting  quality  problems  agreed  upon  by  at  least  two  of  the 
cardiology  reviewers: 


Positive  MD  hrs  RN  hrs 

Predictive        per  detected     per  detected 
Sensitivity        Specificity        Value  problem  problem 


UCDS  85%(+/-16)      48%(+/-12)      25%(+/-ll)      2.1 

JHH  74%(+/-12)     57%(+/-ll)     27  (+/-12)        1.9 

HMPS  69%(+/-16.3)  56%(+/-12)      26(+/-ll)         2.0 


6.7 
1.9 
2.0 


Sensitivity  was  similar  for  all  cohorts  for  UCDS,  and  better  than  JHH  and  HMPS  for  all. 
Sensitivity  for  MI  patients  cannot  be  evaluated  properly  due  to  the  absence  of  quality 
problems  making  it  impossible  to  produce  pooled  estimates  in  that  cohort.  Specificity  of 
JHH  and  HMPS  screens  exceeded  that  of  UCDS  for  PTC  A  and  MI  patients.  Positive 
predictive  values  for  UCDS  were  similar  for  all  cohorts,  and  similar  to  that  of  JHH  and 
HMPS  screens,  except  for  the  MI  cohort,  which  again  due  to  the  absence  of  quality 
problems  in  one  stratum  made  pooled  estimates  difficult  to  interpret.  Because  of  this,  the 
number  of  physician  and  nurse  hours  per  quality  problem  detected  was  greater  with  JHH 
and  HMPS  screens  than  for  UCDS  for  MI  patients  (physician  hours:  JHH  6.3,  HMPS  7.1, 
UCDS  1.9,  nurse  hours:  JHH  9  7,  HMPS  9.6,  UCDS  5.6).  For  all  other  cohorts,  nurse 
hours  per  quality  problem  detected  were  6.7  hours  for  UCDS  and  1-2  hours  for  the  other 
systems,  while  physician  time  per  quality  problem  was  approximately  2  hours  for  all 
systems. 

Conclusion: 

For  the  cardiac  patients  at  this  hospital,  the  version  of  UCDS  tested  was  more  sensitive 
and  less  specific  for  detecting  quality  problems  than  the  other  two  screening  methods,  but 
was  much  less  efficient  per  quality  problem  detected.  There  are  differences  in  sensitivity 
and  specificity  depending  on  whether  patients  were  admitted  for  MI,  angioplasty,  or 
bypass  surgery,  with  UCDS  performing  significantly  better  in  MI  patients  than  the  other 
two  systems.  Positive  predictive  values  were  similar  for  all  three  methods  tested  and  were 
only  about  25%  in  this  population  with  a  prevalence  of  1 8%  quality  problems.   All  systems 


could  be  streamlined  for  these  populations  of  patients  to  include  only  the  highest  yield 
screens  with  little  loss  of  sensitivity. 

BACKGROUND 

Regulatory  efforts  to  monitor  quality  of  care  for  payers,  like  the  Health  Care  Financing 
Administrations's  (HCFA's)  contracts  with  peer  review  organizations  (PROs)  are  moving 
toward  methods  of  screening  for  quality  problems  that  depend  on  aggregate  analyses  of 
processes  or  outcomes  of  care  for  cohorts  of  patients.  For  some  conditions  with  adequate 
numbers  of  patients  per  provider  or  per  hospital,  such  review  can  be  based  upon  risk- 
adjusted  outcomes  assessment.  For  other  high  volume  patient  cohorts,  the  effort  and 
expense  are  justified  to  codify  guidelines  for  care  and  update  them  regularly,  and  for  such 
patients  quality  of  care  is  likely  to  be  monitored  using  review  against  explicit  criteria  based 
on  these  guidelines.  However,  for  most  episodes  of  care,  implicit  review  by  physicians  of 
the  medical  record  or  of  observed  medical  events,  remains,  and  probably  will  remain,  an 
important,  practical  way  to  identify  the  rate  and  type  of  quality  problems  that  have 
occurred.  Although  peer  review  by  one  or  two  physicians  is  not  reliable  enough  to  draw 
conclusions  about  an  individual  case,  it  is  a  sound  method  for  identifying  patterns  and 
rates  of  quality  problems  for  groups  of  patients 

The  HCFA  Health  Standards  and  Quality  Bureau  contracts  with  PROs  to  review 
medical  records  of  hospitalizations  to  identify  patterns  of  quality  problems  experienced  by 
patients  treated  by  a  given  physician  or  hospital.  Hospitals'  internal  efforts  to  assess  and 
improve  the  clinical  quality  of  care  also  depend  heavily  on  peer  review  by  physicians. 
Because  physician  review  is  time-intensive,  screening  systems  to  identify  records  with 
possible  quality  of  care  problems  have  been  evolved  in  an  effort  to  save  physician  review 
time  by  directing  it  to  a  higher  yield  group  of  records.  HCFA  first  attempted  to  give 
guidance  in  the  screening  process  by  disseminating  to  PROs  a  set  of  generic  screens. 
These  mainly  consisted  of  a  set  of  adverse  outcomes.  A  similar  set  of  adverse  outcome 
screens  was  also  developed  by  investigators  conducting  the  Harvard  Medical  Practice 
Study,  who  also  required  a  way  to  screen  records  to  determine  which  required  physician 
review.  More  recently,  HCFA  developed  a  record  abstraction  system  known  as  the 
Uniform  Clinical  Data  Set  (UCDS).  Although  the  UCDS  has  many  other  purposes, 
including  collecting  data  for  risk-adjusted  outcomes  analyses  as  well  as  explicit  review  for 
patterns  of  care,  early  prototypes  of  the  UCDS  software  included  a  set  of  embedded 
algorithms  designed  to  flag  medical  records  that  should  be  reviewed  by  a  physician. 

The  performance  of  screening  systems,  in  terms  of  sensitivity,  specificity, 
predictive  value  and  cost  for  identifying  quality  problems,  has  rarely  been  evaluated. 
Studies  have  demonstrated  the  HCFA  generic  screens  and  the  methods  used  by  one  PRO 
have  low  predictive  value  and  relatively  low  sensitivity  (Sanazaro  and  Mills,  1991,  Rubin 
1992).  It  was  unknown  if  the  set  of  algorithms  developed  for  UCDS  had  higher  sensitivity 
and  predictive  value  and  how  much  better  performance  was  achieved  given  what  appeared 
to  be  a  very  extensive  data  extraction  from  the  medical  record.  In  addition,  the  cost  per 
quality  problem  identified  had  not  been  determined  for  any  of  the  screening  methods 
above. 


PURPOSE  AND  AIMS 

The  purpose  of  this  study  was  to  assess  the  sensitivity,  specificity,  predictive  value  and 
relative  costs  of  three  different  systems  designed  to  screen  medical  records  for  problems 
with  the  quality  of  hospital  care.  The  systems  were  1)  a  set  of  screens  used  by  Johns 
Hopkins  Hospital  that  included  HCFA  generic  screens  used  by  the  PROs,  2)  the  Harvard 
Medical  Practice  Study  (HMPS)  screens,  and  3)  screening  algorithms  embedded  in  version 
3.01  of  the  Uniform  Clinical  Data  Set  (hereafter  referred  to  as  UCDS).  The  reference 
standards  used  to  establish  a  quality  problem  were  a  judgment  of  substandard  ("below 
standard")  care  rendered  at  the  end  of  a  structured  implicit  review  method  developed  at 
RAND  (Rubin  et  al.  1990,1992),  a  standardized  summary  quality  score  obtained  by 
summing  specific  aspects  of  care  rated  in  the  structured  form,  and  a  judgment  of  whether 
or  not  there  was  any  "quality  problem",  determined  after  the  rest  of  the  review  was 
complete.  We  also  used  a  modification  of  the  physician  form  used  to  determine 
negligence  used  by  the  Harvard  Medical  Practice  Study  (Brennan  et  al.   1989),  but  this 
applied  only  to  records  with  adverse  events  and  thus  could  not  be  used  as  a  general 
reference  standard. 

Prior  to  undertaking  the  study,  we  met  with  Health  Standards  and  Quality  Bureau 
staff  at  HCFA  to  determine  in  which  population  it  might  be  most  helpful  to  test  different 
screening  systems.  Staff  at  that  point  in  time  were  trying  to  refine  UCDS  for  various 
purposes  in  patients  with  cardiac  disease,  and  foresaw  its  evolution  into  condition-specific 
data  abstraction  modules.  They  believed  the  results  would  vary  from  condition  to 
condition  and  suggested  beginning  with  cardiovascular  disease  as  HCFA  was  beginning 
with  those  conditions  in  its  efforts  to  evolve  other  forms  of  pattern  and  outcome  analysis 
as  well  Therefore,  this  study  was  limited  to  patients  with  coronary  artery  disease, 
specifically,  those  entering  the  hospital  because  of  a  myocardial  infarction  or  to  undergo 
coronary  angioplasty  or  bypass  surgery. 

Subsidiary  aims  of  the  study  were  to  measure  performance  of  the  screening 
systems  at  different  levels  of  comorbidity  and  severity  of  illness  at  admission,  as  well  as  to 
measure  the  interrater  reliability  of  screening  methods  and  physician  review  as  applied  in  a 
hospital  user's  quality  assurance  program. 

METHODS 
Overview 

A  sample  of  medical  records,  stratified  by  whether  or  not  they  had  passed  the  hospital's 
own  screening  review  (which  included  the  HCFA  generic  screens)  and  also  stratified  by 
whether  they  were  admitted  for  MI,  PTCA  or  CABG,  were  screened  by  trained  nurses 
using  all  three  systems.  Each  medical  record  also  underwent  two  independent  reviews  by 
physicians.  Interrater  agreement  was  assessed  in  a  subsample  of  cases  abstracted  by  two 
nurses,  and  for  the  total  sample  between  the  two  independent  physician  reviews. 
APACHE  III  and  Charlson  scores  were  calculated  for  each  record  as  well. 

Selection  of  Records 

Records  were  from  patients  admitted  to  Johns  Hopkins  Hospital,  an  1 100-bed  urban 
teaching  hospital.  The  hospital  is  a  referral  center  for  PTCA  and  CABG  from  Baltimore, 
other  counties  in  Maryland,  and  the  Washington,  D.C.  area;  approximately  1200  PTCA 


and  800  CABG  operations  are  performed  annually.  Routinely,  quality  assurance  nurses 
review  100%  of  records  using  a  set  of  generic  indicators  that  included  HCFA's  generic 
screens  as  a  full  set  until  mid-1991 .  Records  were  sampled  from  7/1/89  through  6/30/91 . 
CABG  cases  were  defined  as  ICD-9-CM  codes  36. 10-36. 19;  PTCA  cases  as  ICD-9-CM 
codes  36.01,36.02,  36.05;  and  MI  cases  as  ICD-9-CM  diagnoses  codes  410. XI. 

The  sample  within  each  cohort  was  also  stratified  by  whether  or  not  the  record  had 
failed  the  hospital's  own  quality  assurance  screen,  in  order  to  enrich  the  sample  with 
quality  problems.  The  size  of  the  sample  was  determined  by  how  large  a  sample  would  be 
needed  to  estimate  a  true  sensitivity  of  50%  to  within  20%  (other  true  values  of  sensitivity 
would  require  a  smaller  sample  size).  The  sample  sizes  needed  were: 


CABG/Fail 
CABG/Pass 

PTCA/Fail 
PTCA/Pass 

Ml/Fail 
Ml/Pass 


Needed 

True  population 

Sample 

Sam 

281 

72 

177 

1281 

89 

36 

136 

57 

114 

1169 

89 

32 

41 

29 

34 

243 

69 

56 

Total 


3151 


405 


460 


Fifty  records  were  re-abstracted  to  give  us  80%  power  to  detect  at  an  alpha  error  of  .05 
the  difference  between  a  reliability  for  two  groups  of  40%  and  60%,  or  a  difference 
between  reliability  for  two  groups  of  60%  and  75%. 


Record  Screening 

One  of  the  physician  investigators  (L.  Camacho)  underwent  training  at  Wisconsin  PRO  in 
UCDS  version  3.01  record  abstraction.  He  trained  quality  assurance  nurses  at  Johns 
Hopkins  to  abstract  records  using  UCDS  in  the  medical  records  department  of  the 
hospital,  using  laptop  computers.  Data  elements  collected  by  UCDS  include 
sociodemographics,  admitting  and  discharge  information,  medical  history  and  physical 
examination,  laboratory  and  diagnostic  tests,  endoscopy,  operative  episodes,  treatment 
interventions,  and  hospital  course  (adverse  events,  admission  to  special  units,  etc.)  Using 
automated  algorithms,  UCDS  then  generates  a  summary  of  whether  the  case  has  failed  any 
of  its  flags.  The  UCDS  algorithms  are  organized  into  generic  screens,  elective  surgery, 
disease-specific  screens,  organ-specific  screens,  and  discharge  status.   In  each  decision 
area,  algorithms  look  for  a  primary  condition  which  defines  the  cases  and  the  context  in 
which  the  screen  is  applicable.   Then  it  looks  for  a  secondary  condition  that  triggers  a 
specific  flag.  For  example,  if  a  blood  transfusion  was  given  (primary  condition)  and  the 
patient  had  an  unexpected  event  of  shock  due  to  drug  or  transfusion  reaction  within  one 
day  after  the  blood  transfusion  (secondary  condition),  the  following  flag  is  set:   "Shock 


due  to  transfusion  reaction  within  one  day  of  blood  administration".    A  complete  listing  of 
UCDS  flags  is  available  in  accompanying  documentation  (Camacho  1994). 

The  same  nurses  applied  Johns  Hopkins  Hospital  and  Harvard  Medical  Practice 
Study  screens,  which  had  also  been  automated  on  the  same  computers.  Johns  Hopkins 
Hospital  screens  included  HCFA's  set  of  generic  screens.  They  are  events  or  situations 
thought  to  signal  a  potential  for  poor  quality  care.  For  example,  cases  of  unscheduled 
return  to  surgery  on  the  same  hospital  admission  for  the  same  condition  as  the  previous 
surgery  or  to  correct  operative  problems,  would  fail  screening.  The  full  set  of  JHH 
screens  are  described  elsewhere  (Camacho  1994,  Appendix  B)  Harvard  Medical  Practice 
Study  screens  are  sixteen  events  associated  with  potential  poor  quality  care,  overlapping 
somewhat  with  JHH  screens  (Camacho  1994,  Appendix  B). 

Quality  control  for  nurse  abstraction:  All  nurse  abstractions  were  overread  by 
one  of  the  investigators  to  confirm  accuracy  of  identification  variables.  Dr.  Camacho 
abstracted  data  for  comorbidity  and  severity  of  illness  scores,  and  in  the  process  became 
acquainted  with  each  record  well  enough  to  detect  if  inappropriate  flags  had  been 
triggered  in  UCDS  due  to  inaccurate  data  abstraction.  Each  UCDS  printout  was  overread 
and  the  medical  record  checked  to  confirm  aberrant  lab  data  or  diagnostic  tests,  or 
suspicous-looking  data. 

Physician  Review 

Each  record  was  reviewed  by  two  or  three  physicians  using  1 )  a  modification  of  a 
structured  implicit  review  form  for  hospital  care  for  diverse  medical  and  surgical 
conditions  (Rubin  et  al.  1990).  Modifications  consisted  of  adding  items  asking  for  brief 
descriptions  of  any  quality  problems  and  justifications  for  some  answers,  adding  tests  and 
treatments  relevant  to  cardiac  care,  and  excluding  some  items  that  had  not  performed  well 
in  other  settings;  and  2)  a  condensed  form  of  the  adverse  event  analysis  form  from  the 
Harvard  Medical  Practice  Study  (Brennan  et  al.  1989),  eliminating  items  asking  about  the 
organ  system  involved  in  the  adverse  event,  where  it  took  place,  the  responsible  specialty, 
and  others  that  were  not  relevant  to  this  study.  Items  were  retained  that  asked  about 
whether  there  was  injury  (morbidity  not  caused  by  the  disease  process),  adverse  events 
(injuries  caused  by  medical  management),  negligence  (failure  to  meet  the  average 
practitioner's  standard  of  care  at  the  time  care  was  given),  and  scores  for  the  confidence  in 
medical  causation  of  the  adverse  event  and  confidence  that  negligence  occurred.  We 
defined  negligence  or  adverse  event  as  requiring  a  score  of  4  or  higher  on  the  6-point 
scale.  At  the  end,  physicians  were  asked  on  a  "physician  opinion  form"  to  review  the 
screening  results  (previously  sealed)  to  comment  on  whether  any  quality  problems  found 
were  related  to  the  screens  the  case  had  failed,  and  invited  a  critique  of  the  screens  as  they 
worked  in  this  particular  case. 

Determining  reference  standard  from  physician  review 

To  increase  the  reliability  of  reference  standards,  we  defined  "substandard"  care  or  quality 
problem  only  when  two  reviewers  agreed  upon  the  determination.   For  the  summary 
score,  we  used  the  mean  of  the  two  reviews. 

The  summary  score  was  calculated  using  only  items  on  the  structured  implicit 
review  form  that  showed  some  predictive  ability  for  selected  outcomes  (injury,  adverse 


events,  and  disability)  in  a  multiple  regression  model.  The  final  model  had  14  variables  to 
which  weights  were  assigned  according  to  the  strength  of  their  relationship  to  disability  for 
that  patient  cohort..  To  combine  items  with  different  scales,  items  were  scored  as  a  ratio 
of  assigned  to  applicable  points  (RAAP).  Summary  scores  were  also  adjusted  for 
reviewers'  own  mean  by  substracting  the  RAAP  from  that  reviewer's  mean  for  all  cases  he 
had  reviewed,  and  dividing  by  the  standard  deviation  for  the  whole  sample.   A  cutoff 
standardized  score  was  then  determined  that  maximized  agreement  with  determinations  of 
substandard  care,  quality  of  care  problems,  and  negligence. 

In  this  report,  we  emphasize  findings  using  the  reference  standard  of  a  "quality 
problem"  from  the  physician's  opinion  form,  as  this  definition  identifies  opportunities  for 
improvement  even  when  the  care  on  average  was  considered  "standard"  or  "above 
standard"  on  the  implicit  review  form.  Analyses  using  substandard  care  and  the  summary 
score  as  the  reference  standard,  as  well  as  analyses  using  a  quality  problem  judged  by 
either  reviewer  rather  than  both,  are  reported  in  detail  elsewhere  (Camacho  1 994, 
Appendix  D). 

Assessment  of  reliability 

A  detailed  description  of  the  methods  used  to  assess  reliability  is  available 
(Camacho  1994).  In  brief,  we  calculated  crude  ageement,  kappa,  and  two  alternatives  to 
kappa  that  have  been  recently  described:  the  achievement  ratio,  and  kappa-free.  These 
two  statistics  attempt  to  adjust  the  kappa  calculation  for  the  maximum  kappa  achievable 
using  a  particular  distribution,  as  kappa  is  misleading  when  events  are  distributed  in  a 
highly  skewed  fashion,  such  as  quality  problems.  Kappa-free  assumes  that  the  a  priori 
probability  of  a  quality  problem  is  50%  in  a  reviewers  mind,  rather  than  the  distribution 
found  post-hoc. 

In  all  the  reported  analyses,  we  assess  the  reliability  of  one  reviewer's  judgment. 
However,  as  a  reference  standard  in  the  study,  we  used  judgments  agreed  upon  by  two 
reviewers  which  greatly  increases  the  reliability  of  the  findings.  We  also  calculated  the 
estimated  kappa  of  such  two-reviewer  consensus  judgments  but  did  not  make  these 
calculations  using  kappa- free  and  the  achievement  ratio. 

Peformance  of  screening  systems 

Performance  of  screening  systems  against  the  reference  standard  was  assessed 
using  sensitivity  and  specificity,  predictive  value,  and  the  number  of  nurse-hours  and 
physician-hours  per  quality  problem  detected.    For  the  latter,  we  assumed  half  an  hour  per 
record  for  physician  review.  For  screening  systems,  we  used  HCFA's  estimate  of  65 
minutes  for  UCDS  3.01,and  15  minutes  for  JHH  or  HMPS  based  on  our  own  experience 
and  the  quality  assurance  nurses'  experience.  Performance  of  the  screening  system  was 
compared  to  the  sensitivity,  specificity,  predictive  value  and  nurse  and  physician  hours  per 
quality  problem  if  the  same  number  of  records  were  selected  at  random  rather  than 
screened  in  order  to  be  referred  to  physicians. 

All  results,  both  of  analyses  of  interrater  agreement  and  of  screening  system 
performance,  pool  the  results  of  the  two  strata  within  each  condition  using  a  reweighting 
that  reflects  the  original  proportions  of  each  stratum  for  that  condition  as  detailed  in  the 
sample  selection  above.  When  results  are  pooled  across  conditions,  they  are  also  re- 


weighted  to  reflect  the  proportions  of  admissions  with  that  condition  in  the  hospital  during 
the  time  period  examined  in  the  study. 

RESULTS 

Sample 

About  65%  of  records  were  available  for  review;  thus  additional  records  were  requested  at 
random  until  the  appropriate  sample  size  was  reached.  A  total  of  460  records  were 
examined.  There  were  511  nurse  abstractions  and  951  physician  reviews.  Nine  cases 
were  excluded  because  the  principal  procedure  was  wrongly  coded,  the  clinical  condition 
of  interest  was  not  present,  or  there  were  too  many  missing  parts  to  allow  a  review.  28 
cases  were  excluded  because  one  of  the  physician  reviews  did  not  occur.  Fourteen 
percent  of  cases  were  reviewed  by  a  third  reviewer. 

CABG  patients  were  predominantly  white  males  in  their  sixties  or  seventies, 
retired,  covered  by  private  health  insurance  and/or  Medicare,  who  had  had  a  previous 
hospital  admission  for  a  related  condition  in  the  prior  year.  Coronary  atherosclerosis  was 
the  most  common  principal  diagnosis,  they  were  discharged  to  home  in  two  weeks,  on 
average. 

PTC  A  patients  were  typically  white,  slightly  more  likely  to  be  male,  between  55 
and  70,  employed  or  retired  and  capable  of  self-care.  They  were  covered  by  private 
insurance,  Medicare,  or  both,  and  often  had  not  been  hospitalized  previously  for  a  related 
condition.  Their  principal  diagnosis  was  most  likely  to  be  coronary  atherosclerosis,  or, 
less  often,  an  MI,  and  they  were  less  severely  ill  with  less  comorbid  disease  than  CABG 
and  MI  patients.  A  substantial  proportion  had  adverse  occurrences,  but  few  had  intra- 
operative adverse  events  (PTCA  is  defined  as  an  operation  by  UCDS).  Discharge  to  home 
occurred  typically  in  5  days  in  this  time  period  between  1989  and  1991. 

In  the  MI  stratum,  patients  were  typically  white  or  black  males,  aged  40  to  80. 
Most  were  retired  or  employed,  and  were  covered  by  Medicare  or  private  insurance. 
Most  were  capable  of  self-care  before  the  MI,  had  not  been  hospitalized  for  a  related 
condition,  but  were  sicker  at  admission  and  had  more  comorbid  conditions  than  CABG 
and  PTCA  patients. 

These  demographic  and  clinical  characteristics  of  the  sample  are  detailed 
extensively  elsewhere  (Camacho  1994,  pp.  98-120). 

Screening  Results 

UCDS  adverse  occurrences  were  more  frequent  in  the  CABG  stratum  (50%)  than 
in  PTCA  (30%)  or  MI  (30%)  strata.  Pulmonary  edema,  cardiac  arrest,  sustained 
ventricular  tachcardia,  atelectasis,  pneumonia,  and  urinary  retention  were  the  most 
common  adverse  occurrences  (Tables  1  and  2).  Four  patients  had  an  MI  after  admission 
which  became  their  principal  diagnosis,  but  were  also  counted  as  adverse  occurrences. 
The  rate  of  adverse  occurrences  per  100  patient-days  was  5.8  for  CABG,  5.7  for  PTCA 
and  5.0  for  MI.  The  length  of  hospital  stay  increased  with  increasing  numbers  of  adverse 
occurrences. 

Trauma  in  the  hospital  was  concentrated  in  the  CABG  group  (Table  3),  and  was 
rare  in  the  other  strata.  Intra-operative  adverse  events  are  listed  separately  in  Table  4 
We  did  not  include  intraoperative  blood  loss  as  an  adverse  event,  as  UCDS  defines  it  as 
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any  blood  loss  more  than  50  cc  during  an  operative  procedure.  The  most  frequent 
intraoperative  adverse  occurrences  were  sustained  arrhythmia,  myocardial  infarction  and 
hypotension.  Intraoperative  adverse  occurrences  in  UCDS  do  not  all  raise  flags  for  quality 
problems.  Additional  criteria  were  needed  to  raise  a  flag.  Adverse  occurrences  in  UCDS 
should  be  distinguished  from  "adverse  events"  as  defined  in  the  Harvard  Medical  Practice 
Study,  meaning  that  the  event  be  caused  by  medical  management.  UCDS  adverse 
occurrences  are  events  regardless  of  causation. 

Physician  Review:  Interrater  Agreement  and  Results 

One  physician  reviewed  82%  of  the  cases.  Therefore,  agreement  measurements 
obtained  often  reflect  agreement  of  that  reviewer  with  the  other  seven  reviewers.  Chance- 
corrected  agreement  ranged  from  slight  to  moderate  when  examining  this  reviewer's 
agreement  with  other  reveiwers.  A  detailed  analysis  of  individual  reviewers'  tendencies  is 
detailed  elsewhere  (Camacho  1994). 

These  reliability  assessments  are  for  the  reliability  of  one  physician's  review. 
Because  of  poor  reliability,  we  used  as  the  reference  standard  for  the  study  a  quality 
problem  that  both  physicians  agreed  about.  The  estimated  reliability  for  this  is  more  than 
twice  as  high  as  the  reliability  statistics  presented  here  for  interrater  agreement. 

Table  5  indicates  agreement  about  substandard  care,  which  was  79%,  with  kappa 
0.1 1,  and  kappa-free  0.57.  Only  2.8%  of  cases  had  below  standard  care  after  re-weighting 
for  the  stratification  by  cases  that  failed  JHH  screens;  2.5%  for  CABG,  3.2%  for  PTCA 
and  2.0%  for  MI.  Tables  6-8  detail  agreement  about  substandard  care  for  each  condition. 

Table  9  indicates  that  agreement  about  a  "quality  problem"  was  68%,  with  kappa 
0.29,  the  achievement  ratio  (kappa/maximum  achievable  kappa)  0.33,  and  kappa-free 
0.37.  Of  all  cases  pooled  across  all  strata,  17.8%  had  a  quality  problem  agreed  upon  by 
two  reviewers,  while  49.5%  had  a  quality  problem  determined  by  at  least  one  reviewer.. 
Proportions  of  quality  problems  for  the  different  conditions  ranged  from  15%  to  34%  in 
single  reviews,  and  from  4%  to  18%  for  ratings  agreed  upon  by  two  reviewers  (Tables  10- 
12).  Crude  agreement  ranged  from  67%  to  76%,  with  CABG  and  MI  having  poor 
agreement  using  the  traditional  kappa  measure.  Kappa-free  is  a  better  statistic  for  this 
highly  skewed  data,  and  was  0.33.  0.38,  and  0.51  for  CABG,  PTCA,  and  MI  respectively. 

Quality  problems  and  substandard  care  were  closely  associated  (Table  13).    A 
quality  problem  can  be  more  minor,  such  that  care  overall  is  still  judged  standard  or  above. 
Thus  a  quality  problem  agreed  on  by  two  reviewers  occurred  in  1 5  of  1 5  records  that  both 
reviewers  agreed  had  substandard  care,  but  also  in  24  of  359  records  that  both  reviewers 
agreed  had  care  that  was  standard  or  above. 

Nurse  Abstractor  Agreement 

We  examined  agreement  based  on  nurse-abstracted  data,  for  UCDS,  and  based  on 
nurse  application  of  screens,  for  JHH  and  HMPS,  about  whether  or  not  a  case  failed  a 
screen  and  needed  referral  for  physician  review 

Agreement  about  referral  was  98%  for  UCDS  (Table  14),  while  agreement  using 
each  subset  of  UCDS  flags  (the  five  different  types  of  flags  included)  ranged  from  52%  to 
98%.   At  an  individual  screen  level  agreement  was  nearly  perfect  usually  exceeding  95%. 
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For  example,  agreement  about  an  unexpected  event  following  surgery  as  determined  in 
UCDS  was  98%  with  kappa  .95,  kappa-free  0.96  (Table  16).  Agreement  about 
inappropriate  hospitalization  and  services  for  ischemic  heart  disease  and  chest  pain  in 
UCDS  was  96%  with  kappa  .65,  kappa-free  0.92  (Table  17). 

Agreement  for  the  complete  set  of  JHH  screens  was  76%  (Table  17).  However, 
for  most  individual  screens,  chance-corrected  agreement  was  substantial  to  almost  perfect 
(Tables  18-19).  Only  for  one  screen  was  agreement  low  (Table  20). 

Table  21  shows  results  for  the  HMPS  set  of  screens  as  a  whole.  Again,  agreement 
was  higher  for  individual  screens  (Tables  22-23). 

Screening  Results 

Proportions  of  screen  failures  varied  from  68%  to  38%  across  conditions  and 
screening  systems  (Table  24).  A  large  proportion  of  CABG  patients  failed  JHH  and 
HMPS  screens,  while  in  UCDS  the  proportion  was  lower.  For  PTCA  and  MI  the  pattern 
was  the  opposite.  Pooling  all,  the  difference  in  the  proportion  of  screen  failures  was 
similar.  Cases  that  passed  UCDS  but  failed  JHH  and  HMPS  did  so  because  of  "hospital- 
incurred  trauma"  or  "other  undesirable  outcomes",  "complication  of  diagnostic  or  invasive 
procedure",  "abnormal  diagnostic  finding  not  addressed",  and  a  few  other  screens.  For  all 
screening  systems,  patients  who  failed  had  higher  lengths  of  stay,  more  comorbidity  and 
higher  severity  of  illness. 

Table  25  details  frequency  of  UCDS  flags  by  main  decision  areas.  A  high 
proportion  of  cases  were  flagged  for  iatrogenic  events,  failure  to  establish  severity  of 
illness,  insufficiency  of  acute  care  services,  and  unexpected  event  following  surgery.  The 
most  frequent  flags  for  CABG  were  procedure  complications,  unexpected  events  within 
three  days  of  surgery,  severity  of  illness  criteria  not  established  for  calcium  abnormalities 
and  acid-base  disturbances,  and  acute  care  services  not  rendered  for  anemia.  All  the  same 
flags  except  acid-base  disturbances  were  frequent  for  PTCA.  Lack  of  follow-up  of  chest- 
x  ray  abnormalities  and  severity  of  illness  not  established  for  pulmonary  abnormalities 
occurred  in  a  substantial  proportion  of  cases.  Again,  anemia,  calcium  abnormalities, 
pulmonary  abnormalities,  and  lack  of  follow-up  for  chest  x  ray  abnormalities  were  among 
the  most  frequent  for  MI,  as  well  as  unexpected  death.  Extensive  details  are  given 
elsewhere  on  the  frequency  of  inidividual  UCDS,  JHH  and  HMPS  flags  (Camacho  1994, 
Appendix  D). 

The  most  frequent  JHH  screens  failed  were  abnormal  diagnostic  findings  not 
addressed,  hospital-acquired  infections,  and  complications  of  diagnostic  or  invasive 
procedures. 

The  most  frequent  HMPS  screens  failed  were  for  subsequent  admissions  for  the 
same  condition,  hospital-incurred  trauma,  cardiac-respiratory  arrest  (CABG  and  MI), 
death  (CABG  and  MI)  and  "other". 

Screening  System  Performance 

Table  26  indicates  the  performance  of  the  three  screening  systems  in  each  of  the 
three  conditions  and  for  the  records  pooled,  reweighted  according  to  the  representation  of 
these  conditions  at  Johns  Hopkins.  UCDS  was  most  sensitive  compared  to  the  other 
screening  systems  for  PTCA  and  MI,  although  confidence  intervals  overlap  widely.  JHH 
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screens  were  more  sensitive  for  CABG.  The  "pass"  stratum  in  the  MI  cohort  had  no 
quality  problems  so  that  a  pooled  sensitivity  is  misleading;  there  was  no  variance  between 
the  strata).  The  sensitivity  was  high  for  the  other  stratum  which  had  quality  problems. 

The  specificity  was  lower,  less  than  60%  in  most  cases.  Predictive  values  were 
similar  across  screening  systems  except  in  the  MI  stratum,  which  had  a  small  number  of 
confirmed  quality  problems.    For  most  screening  systems  only  25%  of  cases  failing 
screens  had  confirmed  quality  problems.  This  is  not  much  better  than  the  1 8%  of  cases 
that  would  have  confirmed  quality  problems  if  records  were  selected  for  review  at  random. 

On  average,  it  took  6.7  hours  of  record  abstraction  with  UCDS  to  detect  one 
quality  problem,  and  2  hours  of  nurse  screening  with  JHH  or  HMPS  criteria. 

Performance  of  Screen  Subsets 

The  contribution  of  each  screen  was  assessed  by  deleting  it  from  the  entire  set  and 
comparing  the  performance  of  the  remaining  subset  with  the  complete  set.  Few  flags  had 
a  measurable  effect  on  the  performance  of  the  UCDS  system  as  a  whole  (Table  27).  Six  of 
the  UCDS  flags  were  two  or  more  times  more  likely  to  be  found  in  cases  with  a  quality 
problem,  whereas  four  were  more  likely  to  be  found  in  cases  without  a  quality  problem. 
The  effect  of  some  screens  varied  across  conditions,  and  these  are  detailed  elsewhere 
(Camacho  1994,  Appendix  D,  Tables  D23  to  D25)..  Withdrawing  flags  serially  (Figure  1) 
caused  UCDS  screening  to  become  less  sensitive  and  more  specific  as  expected.  Two 
flags  were  very  important  to  maintaining  the  sensitivity  of  UCDS  to  quality  problems  for 
these  conditions.  These  were  "diagnosis  codes  include  complications  of  procedures"  and 
"acute  care  services  not  rendered  for  anemia".  Other  flags  caused  only  small  changes  in 
accuracy.  Manipulating  the  screen  set  did  not  improve  the  positive  predictive  value  much. 

For  JHH  screens,  five  of  the  screens  showed  measurable  effects  on  the  efficiency 
and  performance  of  the  JHH  system  (Screens  1025,  154,  101,  155,  156,  see  Table  28). 
The  most  frequent  flags  are  also  the  ones  with  the  lowest  yield  and  low  predictive  value 
with  likelihood  ratio  close  to  1 .  Screens  1 55  and  156,  "IV  fluids  after  midnight  on  day  of 
discharge"  and  "purulent  or  blood  draining  or  wound  or  open  area"  occurred  in  only  1.5% 
of  cases  but  had  high  likelihood  ratios.  Three  screens,  "reintubation  within  72  hours  of 
extubation",  "patient  discontinuous  of  lines  while  in  the  SICU",  and  "hospital-acquired 
decubitus  ulcer"  were  found  more  often  in  cases  without  a  quality  problem  (likelihood 
ratio  <  1).  Noteworthy  is  the  high  performance  of  screen  109,  "care  or  lack  of  care 
resulting  in  serious  complications"  which  was  a  very  subjective  screen  applied  by  quality 
assurance  nurses.  For  JHH  screens,  the  plot  portraying  changes  as  screens  were  serially 
withdrawn  (Figure  2)  showed  a  decrease  in  sensitivity  and  increase  in  specificity  with  each 
point  representing  a  subset  without  one  or  more  screens.  Maximum  accuracy  was 
obtained  with  the  complete  set  of  screens.  However,  removing  the  screens  with  poor 
performance  did  not  improve  the  accuracy  of  the  entire  system  very  much.  Details  of  this 
analysis  can  be  found  elsewhere  (Camacho  1994,  Appendix  D,  Table  D29) 

For  HMPS,  three  screens  (2,3,  and  15)  showed  substantial  impact  when  excluded 
from  the  whole  set  of  HMPS  screens  (Table  29).  Most  important  were  "subsequent 
admission  related  to  index  admission",  and  "other  undesirable  outcome".  "Hospital- 
incurred  trauma  not  resolved,  or  extending  hospitalization",  also  had  a  small  adverse  effect 
on  sensitivity.  Again,  six  screens  occurred  more  frequently  in  cases  without  a  quality 
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problem.  As  with  other  screening  systems,  the  plot  of  sensitivity  as  a  function  of  the  false- 
positive  rate  did  not  show  substantial  improvement  of  the  subsets  with  the  remaining 
screens  (Figure  3).  The  likelihood  ratio  peaks  at  1.7  after  removing  the  screens  with  the 
lowest  predictive  value  (Camacho  1994,  Appendix  D,  Table  D-32). 

Influence  of  Comorbidity  and  Severity 

Table  30  illustrates  that  screening  results  differed  for  different  levels  of  severity  of 
illness,  but  sample  sizes  were  too  small  to  place  much  confidence  in  these  results.  In 
addition,  the  associations  with  severity  and  comorbidity  differed  by  condition.  In  general, 
all  screening  systems  had  higher  sensitivity  in  more  severe  cases,  and  higher  specificity  in 
more  severe  cases.  There  were  only  minor  differences  in  predictive  value  with  level  of 
severity.    As  far  as  comorbidity,  HMPS  and  JHH  systems  were  more  sensitive  in  cases 
with  higher  comorbidity,  whereas  UCDS  sensitivity  was  not  affected  by  comorbidity.  For 
all  screening  systems,  specificity  was  higher  when  comorbidity  was  lower.  Predictive 
value  of  JHH  and  HMPS  were  higher  among  cases  with  lower  comorbidity,  while  for 
UCDS  differences  were  very  small. 

Performance  of  screening  systems  and  screens  expressed  using  adjusted  odds  ratios 

Adjusting  for  comorbidity  and  severity,  one  can  determine  the  odds  ratios  for  cases  that 
failed  each  screening  system  relative  to  those  that  passed  (Table  31).  Quality  problems 
were  4.8  times  more  likely  among  records  that  failed  UCDS  than  those  that  passed  (Table 
29).    The  equivalent  numbers  are  2.6  for  JHH  screening,  and  2.5  for  HMPS.  Because  of 
the  sample  size,  these  latter  two  cannot  be  distinguished  from  1 . 

Tables  32-34  give  adjusted  odds  ratios  for  individual  UCDS,  JHH  and  HMPS 
screens.  The  subsets  of  screens  that  gave  the  best  prediction  of  quality  problems  had  eight 
screens  for  UCDS  (Table  32).  Two  of  these  screens  actually  had  a  negative  association 
with  a  quality  problem,  "unexpected  inpatient  event  within  3  days  of  surgery  or 
endoscopy"  and  "decubitus  ulcer  acquired  in  the  hospital".  Five  of  the  screens  belong  to 
the  "iatrogenic  events"  module  of  UCDS  algorithms.  To  be  clear  on  what  this  means,  it 
means  that  system  performance  is  best  when  records  with  these  screens  are  actually 
removed  from  those  referred  for  physician  review  regardless  of  whether  they  are  flagged 
for  any  other  reason. 

Six  JHH  screens  showed  significant  associations  with  quality  problems  (Table  35). 
Three  were  related  to  patient  stability  at  discharge,  one  flagged  unexpected  deaths, 
another  was  related  to  medication  error  or  drug  transfusion  reaction,  and  one  was 
subjective  impression  of  a  quality  problem  by  the  nurse. 

Among  the  five  HMPS  screens  comprising  the  best  set  selected  (Table  36) ,  three 
had  a  negative  association  with  a  quality  problem  ("unplanned  return  to  the  OR",  "hospital 
acquired  neurological  deficit",  "cardiac  or  respiratory  arrest"). 

CONCLUSIONS  AND  RECOMMENDATIONS 
Reliability  of  Screening 

Nurse-applied  screening  and  data  abstraction  was  highly  reliable  as  applied  in  this  hospital, 
especially  with  respect  to  individual  screens  but  also  with  respect  to  the  overall  decision 
about  whether  to  refer  a  record  to  a  physician  for  review. 
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Reliability  of  Physician  Review 

As  reported  in  many  previous  studies,  individual  record  review  by  physicians  was 
unreliable.  Reliability  in  this  study  was  even  lower  than  previously  achieved  by  Rubenstein 
et  al.  (1990)  or  Rubin  et  al.  (1992)  using  a  similar  review  tool,  perhaps  due  to  less  training 
time  of  reviewers  and  the  relative  inexperience  of  the  cardiology  reviewers,  who  had  never 
performed  peer  review  before,  unlike  previous  studies  that  utilized  some  seasoned 
reviewers.    At  the  level  of  reliability  achieved,  we  believe  that  even  for  group 
comparisons  such  as  in  this  study,  a  consensus  judgment  of  two  reviewers  should  be  used 
as  a  reference  standard  given  the  low  reliability  of  the  individual  physician's  judgment. 

Quality  Problems  vs.  Substandard  Care 

We  confirmed  that  records  judged  as  substandard  by  reviewers  are  a  small  subset 
of  those  in  which  a  quality  problem  occur.  When  the  technology  of  structured  peer  review 
is  used  for  quality  improvement,  as  we  believe  it  is  best  used,  more  opportunities  may  be 
identified  if  reviewers  are  asked  to  identify  quality  problems  rather  than  substandard  care. 
Due  to  this  finding,  we  changed  from  our  intended  reference  standard  of  substandard  care 
to  a  reference  standard  of  a  "quality  problem"  for  this  report. 

Performance  of  Outcome-based  Generic  Screens 

A  few  outcome-based  generic  screens  perform  surprisingly  well  and  this  subset  of 
screens  may  be  valuable  to  use  for  cardiac  patients.  For  the  two  systems  as  a  whole 
evaluated  here  (JHH  and  HMPS),  the  predictive  value  was  low  (25%)  and  the  sensitivity 
in  the  range  of  60%.  It  cost  about  2  hours  of  nurse  review  and  2  hours  of  physician 
review  per  quality  problem  detected.  Several  screens  could  be  eliminated  that  do  not 
significantly  predict  quality  problems,  This  would  reduce  the  nurse  review  time  slightly, 
although  not  substantially  as  many  of  the  screens  that  worked  well  were  not  specific 
events  but  rather  global  judgments  by  nurses  covering  the  entire  hospital  stay  that  require 
a  complete  record  review  by  the  nurse.  For  example,  predictive  screens  in  the  JHH  system 
included  "medication  error"  and  the  nurse's  subjective  impression  of  a  quality  problem,  and 
predictive  HMPS  screens  included  "other  undesirable  outcome".  Screens  that  predict  the 
lack  of  a  quality  problem  that  are  currently  being  used  in  the  opposite  way  in  both  systems 
should  be  eliminated.  Our  findings  are  based  only  on  one  hospital,  and  should  be 
confirmed  in  other  settings. 

A  good  argument  can  be  made  for  random  selection  of  records  for  physician 
review.  At  the  prevalence  of  quality  problems  in  this  particular  study,  with  random 
selection  of  records,  there  would  be  no  nurse  hours  and  from  two  and  a  half  to  three  hours 
of  physician  review  per  quality  problem  detected.    Depending  on  the  relative  salaries  of 
nurses  and  physicians  in  a  hospital,  it  may  be  cost-effective  compared  to  generic  screening 
to  select  records  at  random  in  order  to  find  the  same  number  of  quality  problems. 

Performance  of  UCDS  version  3.01  screens 

While  UCDS  has  many  other  purposes,  including  the  collection  of  data  for  patterns 
of  process  use  and  appropriateness  and  comparisons  of  risk-adjusted  outcome  in  different 
centers,  our  study  evaluated  the  usefulness  of  the  UCDS  algorithms  as  screening  tools  for 
quality  problems  at  the  level  of  the  hospital  user.  For  this  purpose,  UCDS  was  highly 
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sensitive  to  quality  problems  in  the  cohorts  studied  but  had  the  same  poor  predictive  value 
as  generic  outcome  screening  systems,  referring  many  records  with  no  quality  problems. 
UCDS  had  much  higher  cost  per  quality  problem  detected.  From  the  hospital 
administrators  point  of  view,  UCDS  certainly  would  not  be  cost-effective  for  identifying 
quality  improvement  opportunities  as  a  screening  tool  for  peer  review.  From  a  PRO  point 
of  view,  the  additional  cost  of  UCDS  also  does  not  justify  its  use,  as  100%  record  review 
by  physicians  could  be  conducted  at  the  same  cost  as  UCDS  abstraction  and  subsequent 
physician  review  of  flagged  cases.  However,  UCDS  may  be  cost-effective  if  used  to 
report  outcome  or  process  patterns  compared  to  other  hospitals.  This  remains  to  be 
evaluated  in  other  studies  by  HCFA. 

We  advocate  that  our  study  findings  be  confirmed  in  other  settings,  and  used  to 
eliminate  ineffective  algorithms  and  those  that  have  negative  predictive  value  for  quality 
problems  from  the  UCDS  system  as  employed  for  cardiac  patients. 

A  complete  listing  of  quality  problems  detected  is  attached  (see  Appendix  A). 
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Table  1 


Percent  distribution  within  condition  of  UCDS'  adverse  occurrences3  experienced  during  the 
hospitalization.   Quality  of  Care  Screening  Study.   Baltimore  MD,  1992/1993 


Adverse  Occurrence 

CABG 

MI 

PTCA 

All  Conditions 

N  =  215 

N=90 

N=146 

(%  ±  1.96  S.E.) 

Deep  Vein  Thrombosis 

0.4 

0 

0.3 

0.3±0.1 

Pulmonary  Embolism 

0.2 

1.5 

0 

0.2+0.2 

Wound  Dehiscence 

0.4 

0 

0.1 

0.2+0.1 

Hypovolemic  Shock 

0.1 

0.4 

0 

0.1  ±0.1 

Cardiogenic  Shock 

0.7 

0 

0.2 

0.4+0.2 

Septicemic  Shock 

0.9 

0.4 

0.5 

0.7+0.2 

Shock  due  to  Drug/ 

Transfusion  Reaction 

0.2 

0 

0 

0.1  ±0.1 

Cardiac  Arrest 

5.2 

11.1 

0.8 

3.9±2.1 

Hemorrhage  Requiring 

Drainage 

4.1 

0.4 

0.5 

2.2±2.1 

Pulmonary  Edema 

5.5 

6.2 

6.5 

6.0±4.2 

Sustained  Ventricular 

Tachycardia 

3.5 

3.2 

3.2 

3.3±3.0 

Wound  Infection/ Abscess 

1.7 

1.5 

0.3 

1.1  ±0.3 

Myocardial  Infarction 

1.0 

2.8 

1.1 

1.2±1.0 

Cerebrovascular  Accident 

1.8 

0 

0.2 

1.0±0.2 

Respiratory  Failure 

1.5 

2.4 

0.1 

1.0±0.3 

Urinary  Retention 

4.8 

0 

0.2 

2.5  ±0.3 

Atelectasis 

5.2 

0 

0.2 

2.7±2.8 

Pneumonia 

6.1 

3.9 

0.3 

3.5±2.9 

Other 

33.9 

11.9 

19.8 

26.1  ±7.4 

"  Intra-operative  adverse  occurrences  are  presented  separately 


Table  2 


Percent  distribution  within  condition  of  the  number  of  UCDS'  adverse  occurrences3 
experienced  during  the  hospitalization.    Quality  of  Care  Screening  Study.    Baltimore 
MD,  1992/1993 


#  of  Adverse 
Occurrences 

CABG 

N  =  215 

MI 

N=90 

PTCA 

N=146 

All  Conditions 

(%  ±  1.96 
S.E.)    N=451 

0 

50.9 

70.2 

70.8 

62.6±8.0 

1 

34.2 

19.0 

16.8 

25.6±7.4 

: 

9.7 

6.2 

7.9 

8.6±4.2 

3 

1.2 

3.7 

0.4 

1.1±0.3 

4 

0.8 

0.8 

0 

0.5±0.2 

>  5 

3.1 

0 

0.1 

1.6  ±2.0 

2         _ 


X',4)  =  25.039;  p  <  0.000 
*  Intra-operative  adverse  occurrences  are  not  included. 


Table  3 


Percentage  of  cases  within  condition  with  trauma  (UCDS  database)  suffered  in  hospital 
.  Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993. 


Type  of 
Trauma 


CABG 

MI 

PTCA 

All  Conditions 

N  =  215 

N=90 

N=146 

(%  ±1.96SE.1 
N=451 

Fall  or  Accidental  Injury  4.9  0 

Decubitus  Ulcer  (In  Hospital)  2.5  0.4 

Control  of  Function  Loss  0.2  0 

Injury  with  Etiology  Unknown  0. 1  0 


0.1 

2.5±2.8 

0.1 

1.3  ±0.3 

0 

1.0±0.1 

0 

0.0±0.0 

Table  4 


lures 


Percentage  of  intra-operative  adverse  occurrences  among  operative  procedi 
performed  (UCDS  database),  within  condition.   Quality  of  Care  Screening  Study. 
Baltimore  MD,  1992/1993 


Adverse  Occurrence  CABG         MI  PTCA '       All  Conditions 

N«=307  N*=U  N»=172  N*=500 


Hypotension 

Myocardial  Infarction 

Seizure 

Sustained  Arrhythmia 

Cardiac  Arrest 

Perforate/Excise 

Stroke 

Return  to  O.R.  b 


0.60 

0 

3.00 

1.54 

0.07 

0 

3.40 

1.45 

0 

0 

0.07 

0.03 

4.94 

0 

0.69 

2.74 

0.74 

0 

0.56 

0.60 

0.13 

0 

0.35 

0.21 

0.47 

0 

0 

0.23 

3.15 

2.06 

0.56 

1.98 

'  PTCA  is  counted  as  an  operative  procedure  in  UCDS  database. 

'  Patient  needed  further  surgery  as  a  result  of  complications  arising  from  a  previous 

operative  episode  within  the  same  hospitalization.  Return  to  the  O.R.  is  included 

in  the  total. 

N*  :  Number  of  operative  episodes. 


Table  5 

Percentage  of  ALL  cases  found  to  have  substandard  care  in  two  independent  physician 
reviews  of  medicai  records.  Quality  of  Care  Screening  Study.  Baltimore  MD. 

1992/1993 

Review    1 


Below  Std. 

Std. 

Total 

Below  Std. 

2.8 

7.3 

10.1 

Review  2 

Std. 

14.0 

75.9 

89.9 

Total 

16.8 

73.2 

100.0 

Crude  Agreement:  79.0%  ;    Kappa:   0.11  ±  0.09  ;   xJk^  =  0.11  /  0.71  =  0.15  ; 
Kf„  =  0.57  ;  Reliability  of  substandard  care  agreed  upon  by  two  reviewers:  k  =  0.20 


Table  6 


Percentage  of  CABG  cases  found  to  have  substandard  care  in  two  independent 
physician  reviews  of  medical  records.  Quality  of  Care  Screening  Study.  Baltimore  MD, 

1992/1993 


Review  1 

Below  Std. 

Std. 

Total 

Below  Std. 

2.5 

3.5 

6.0 

Review  2 

Std. 

11.9 

82.1 

94.0 

Total 

14.4 

85.6 

100.0 

Crude  Agreement:  84.6%  ;   Kappa:  0.18  ±  0.12  ;  k/k,^  =  0.18  /  0.55  =  0.32  ;  k^. 
=  0.69  ;  Reliability  of  substandard  care  agreed  upon  by  two  reviewers:  k  =  0.31 


Table  7 

Percentage  of  PTCA  cases  found  to  have  substandard  care  in  two  independent  physician 
reviews  of  medical  records.  Quality  of  Care  Screenine  Studv.  Baltimore  MD 
1992/1993 


Review 

1 



Below  Std. 

Std. 

Total 

Review  2 

Below  Std. 
Std. 

Total 

3.2 
17.8 
21.0 

12.2 
66.8 
89.0 

15.4 
84.6 
100.0 

Crude  Agreement:  70.0%  ;    Kappa:  -  0.001  ±  0.16  ; 

k/k^  =-  0.001  /  0.813  =  0. 123  ;  Kfrte  =  0.4  ;   Reliability  of  substandard  care  agreed 
upon  by  two  reviewers:  k  =  -  0.002 


Table  8 


Percentage  of  MI  cases  found  to  have  substandard  care  in  two  independent  physician 
reviews  of  medical  records.  Quality  of  Care  Screening  Study.  Baltimore  MD. 


1992/1993 

Review 

t    1 

Below  Std. 

Std. 

Total 

Review  2 

Below  Std. 
Std. 
Total 

2.0 

7.1 
9.1 

3.7 
87.2 
90.9 

5.7 
94.3 
100.0 

Crude  Agreement:  89.3%  ;   Kappa:  0.23  ±  0.20  ;   k/k_  =  0.23  /  0.75  =  0.31  :  *,„ 
-  0.78  ;  Reliability  of  substandard  care  agreed  upon  by  two  reviewers:  «  =  0.37 


Table  9 


Percentage  of  cases  with  a  quality  problem  (QP)*  according  to  two  reviews  of  all  cases. 
Quality  of  Care  Screening  Study.    Baltimore  MD,  1992/1993 


Review  1 

QP 

17.8 
18.3 
36.1 

NoQP 
13.4 
50.5 
63.9 

Total 

Review  2 

QP 

NoQP 
Total 

31.2 
68.8 
100.0 

Defined  as  failure  to  meet  the  average  practitioner's  standard  of 


care. 


Crude  Agreement:  68.3%    ;   Kappa:  0.29  ±0.10   ;   k/k^  =  0.29  /  0.89  =  0.33  ; 
Kfree  =  0.37  ;   Reliability  of  quality  problem  agreed  upon  by  two  reviewers:  k  =  0.45 


Table  10 


Percentage  of  cases  with  a  quality  problem  (QP)  according  to  two  rev.ews  of  CABG 
cases.  Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Review  1 


QP  No  QP  Total 


Review  2 


Qp  10.3  12.3  22.6 

NoQP  21.0  56.4  77.4 

Total  31.3  68.7  100.0 


Crude  Agreement:  66.7%   ;   Kappa:  0.16  ±0.13  ;  k/k^  =  0.16  /  0.78  =  0.21  ; 
*Tn.  =  0.33  ;   Reliability  of  quality  problem  agreed  upon  by  two  reviewers:  k  =  0.28 


Table  1 1 

Percentage  of  cases  with  a  quality  problem  <QP)  found  in  two  reviews  of  PTCA 

cases.  Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Rev 

lew  1 

QP 

NoQP 

Total 

QP 

18.3 

15.2 

33.5 

Review  2 

NoQP 

16.0 

50.5 

66.5 

Total 

34.3 

65.7 

100.0 

Crude  Agreement:  68.8%     ;    Kappa:   0.30  ±  0.16  ;   k/k^  =  0.30  /  0.98  =  0.31  ; 
k,to  =  0.38  ;   Reliability  of  quality  problem  agreed  upon  by  two  reviewers:  k  =  0.46 


Table  12 


Percentage  of  cases  with  a  quality  problem  (QP)  according  to  two  reviews  of  MI  cases. 
Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Review  I 

QP 

NoQP 

Total 

QP 

4.2 

10.8 

15.0 

Review  2 

NoQP 

13.5 

71.5 

85.0 

Total 

17.7 

82.3 

100.0 

Crude  Agreement:  75.7%  ;    Kappa:   0.11  ±  0.21;   k/k^  =0.11/  0.90  =  0.12  ; 
Kfne  =  0.51  ;   Reliability  of  quality  problem  agreed  upon  by  two  reviewers:  k  =  0.20 


Table  13 


Substandard  care  vs.  quality  problem  assessed  by  two  physician-reviewers.   Quality  of 
Care  Screening  Study.    Baltimore  MD,  1992/1993 


Oualitv  Prnhl^m 

Results  of  two 

reviews 

Present         Uncertain 

Absent 

TOTAL 

combined 

Present 

15 

0 

0 

15 

Substandard 
Care 

Uncertain 

17 

29 

3 

49 

Absent 

24 

72 

263 

359 

TOTAL 

56 

101 

266 

423' 

Spearman  Coefficient  Correlation:   0.529  (p=0.037) 

•  The  number  differs  from  the  ,0,al  samp.e  of  460  because  nine  records  were 
unavai.abie  for  review  and  28  were  reviewed  by  only  one  pb.vs.cian. 


Table  14 


Agreement  between  Nurse-abstractors.  UCDS  flags  raised  by  all  decision  areas.  Quality 
of  Care  Screening  Study.  Baltimore  MD.  1992/1993 


Abstractor  1 

Flag 

'PA' 

'MO' 

Absent 

TOTAL 

Abstractor 

'PA' 

42 

0 

0 

42 

2 

MO' 

0 

5 

0 

5 

Absent 

1 

0 

3 

4 

TOTAL 

43 

5 

3 

51 

'PA'  :  Refer  case  to  Physician  Advisor.  'MO':  Possible  problems.  Monitoring  of 
patterns  warranted.  Absent:  the  case  passed  screening 

Crude  Agreement:  98.0%  ;   Kappa:  0.93  ±0.13  ;   k/km  =  0.93  /  0.93  =  1.0  ; 
Kfl-  =  0.97 


Table  15 


Agreement  between  Nurse-abstractors.  Flag  GS05PAI1  (Unexpected  event  following 
surgery).  Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Abstractor  1 

GS05PAI1 

Present 

Absent 

TOTAL 

Abstractor 

Present 

15 

0 

15 

2 

Absent 

1 

34 

35 

TOTAL 

16 

34 

50 

Crude  Agreement:  98.0%  ;    Kappa:  0.95  ±  0.08  ;      k/k      =  0.95  /  0.95  = 


1.00; 


"a.  -  0-96 


Table  16 


Agreement  between  Nurse-abstractors.   Flag  DP01PR8D  (Ischemic  Heart 
DiseaseyChest  Pain.  Appropriate  Hospitalization  and  Services).  Quality  of  Care 
Screening  Study.  Baltimore  MD,  1992/1993 


Abstractor  1 

DP01PR8D 

Present 

Absent 

TOTAL 

Abstractor 

Present 

47 

1 

48 

2 

Absent 

1 

1 

2 

TOTAL 

48 

2 

50 

Crude  Agreement:  96.1%  ;    Kappa:  0.65  ±  0.27   ;      k/k_  =  0.65  /  1.00  =  0.65  ; 
*«  =  0.92 


Table  17 


Agreement  between  Nurse-abstractors.   Whole  set  of  JHH  Screens.  Quality  of  Care 
Screening  Study.  Baltimore  MD,  1992/1993 


Abstractor  1 

Fail 

Pass 

TOTAL 

Abstractor 

Fail 

37 

6 

43 

2 

Pass 

6 

2 

8 

TOTAL 

43 

8 

51 

Crude  Agreement:  76.5%  ;    Kappa:  0.11  ±  0.27  ;     k/k^  =  0.11  /  1.00  =  0.11  ; 
Kfree  =  0.53 


Table  18 


Agreement  between  Nurse-ahstmrmrc    r~„    ■ 

urse  abstractors.   Generic  screen:   'Hospital-Acquired  Infection. 

Quality  of  Care  Screening  Study.   Baltimore  MD,  1992/1993 


Abstraqnr  1 

Fail 

Pass 

TOTAL 

Abstractor 

Fail 

7 

2 

9 

2 

Pass 

2 

40 

42 

. 

TOTAL 

9 

42 

51 

Crude  Agreement:  92.2%  ;    Kappa:  0.73  ±  0.27  ;      */ 
"r«  =  0.84 


«m«  =0.73/  1.00  =  0.73  ; 


Table  19 


Agreement  between  Nurse-abstractors.   Genenc  Screen:  'Intravenous  fluids  or  drugs 
after  12  midnight  on  day  of  discharge'.  Quality  of  Care  Screening  Studv.  Baltimore 
MD.  1992/1993 


Abstractor        Fail 
2  Pass 

TOTAL 


Abstractor  I 
Fail  Pass 


3 

1 
4 


3 

44 
47 


TOTAL 


6 
45 

51 


Cr.de  Agreement:  92.2%  ;   Kappa:  0.56  ±  0.27  ;     „«_  =  0.56  /  0.78  =  0.72  ; 
S-  =  0.84 


Table  20 


Agreement  between  Nurse-abstractors.   Generic  screen:  'Abnormal  diagnosuc  finding 
not  addressed  and  resolved  where  the  records  do  not  explain  why  they  are  not 
resoivea'.    Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Abstractor  I 
Fail         Pass         Unknown1        TOTAL 


Abstractor       Faii  28 

2  Pass  4 

TOTAL  32 


16 


1 
0 

1 


45 

6 

51 


'  Documentation  was  considered  insufficient.  This  category  was  included  to  compute 
Kappa. 


Crude  Agreement:  58.8%  ;   Kappa:  -0.03  ±  0.22  ;    k/k^,  =  0.03  /  0.37  =  0.42  ; 
"A.  =  0.20 


Table  21 


Agreement  between  Nurse-abstractors.   HMPS  Screening  Criteria:   Whole  set.   Quality 
of  Care  Screening  Study.   Baltimore  MD,  1992/1993 


Abstractor  1 


Abstractor        Fail 


Fail  Pass 


39 


TOTAL 


42 


Pass 


10 


TOTAL  44 


52 


Crude  Agreement:  84.6%  ;   Kappa:  0.46  ±  0.27  ;   k/k^  =  0.46  /  0.87  =  0.54 
•W  =  0.69 


Table  22 

Agreement  between  Nurse-abstractors.   HMPS  Screening  Criterion: 'Death,  except  (1) 
m  planned  terminal  care,  (2)  DNR  order,  and  (3)  patient  is  gravely  ill  from  a  major 
disease  process  (massive  trauma,  thermal  burns  >  50%,  etc.)'.   Quality  of  Care 
Screening  Study.  Baltimore  MD,  1992/1993 


Abstractor 


Abstractor 
Fail  Pass 


TOTAL 


Fail 

8 

2 

10 

Pass 

1 

41 

42 

TOTAL 

9 

43 

52 

Crude  Agreement:  93.8%  ;    Kappa:  0.81  ±  0.27  ;   k/k_  =  0.81  /  0.94  =  0.86  ; 


Kf«  =  0.88 


Table  23 

Agreement  between  Nurse-abstractors.   HMPS  Screening  Criterion:  'Cardiac  or 
respiratory  arrest,  except  (1)  in  planned  teminal  care,  and  (2)  when  patient  is  gravely  ill 
from  a  major  disease  process  (massive  trauma,  thermal  burns  >  50%,  etc.)'. 
Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Abstractor  1 

Fail 

Pass 

TOTAL 

Abstractor 

Fail 

6 

0 

6 

2 

Pass 

3 

43 

46 

TOTAL 

9 

43 

52 

Crude  Agreement:  94.2%  ;    Kappa:  0.77  ±  0.27  ;    K/r       =  0.77  /  0.77  =  1.00 


^frw 


=  0.88 


Table  24 


Percentage  (±1.96  S.E.)  of  screen  failures  in  each  screening  system,  within  each 
condition.  Quality  of  Care  Screening  Study.  Baltimore  MD.  1992/1993 


Screening       CABG  MI  PTCA  All  Conditions 

System  n  =  215  n=9o  N=i46  n=45i 


UCDS  54.6±12.9       65.4±9.6         68.0±14.5       61.1  ±8.8 

JHH  67.6±12.4       37.6±8.9         40.4±14.5       53.6±8.6 

HMPS  63.0±12.7       46.2±9.7         46.3±15.0      54.6±8.9 


Table  25 


Frequency  *  of  UCDS  nags  by  main  decision  areas  within  conditions.   Quality  of  Care 
Screening  Study.  Baltimore  MD,  1992/1993 


Decision  Area 

CABG 

n[%] 

16  [3.7] 

PTCA 
n[%] 

12  [  2.0] 

MI 

n[%] 

All  Conditions 

% 

Discharge  Status/Disposition 

10  [10.9] 

3.3 

Medical  Stability  at  Discharge 

24  [4.5] 

26  [18.6] 

11  [11.3] 

11.0 

Organ  Specific  A  b 

60  [24.6] 

39  [20.7] 

30  [30.4] 

22.7 

Organ  Specific  B  c 

19  [10.2] 

38  [41.4] 

42  [48.7] 

26.6 

Unexpected  Event  Following 
Surgery 

65  [16.9] 

77  [20.6] 

3  [  2.4] 

17.1 

Death 

37  [5.8] 

8  [  0.7] 

19  [  9.2] 

4.0 

Iatrogenic  Events 

155  [26.0] 

50  [12.7] 

12  [  6.2] 

18.7 

cases 


a  Proportions  of  cases  flagged  are  pooled  across  strata.   Percentages  do  not  add  to  the 
total  for  each  condition  since  many  cases  have  more  than  one  flag.   The  number  of 
is  simply  the  sum  of  cases  flagged  disregarding  sample  fractions.   b  Severity  of  illness  not 
established.   c  Acute  care  services  not  rendered. 


Table  26 


Performance  of  Screening  Systems  in  a  set  of  medical  records  with  17.8  %  found  to  have  a 
Quality  Problem  (QP)  by  two  physicians.  Quality  of  Care  Screening  Study.  Baltimore  MD, 


1992/1993 

Screens 

%  of 

Records 

Positive 

Sensitivity 

(%± 

1.96  S.E.) 

Specificity 
(%± 

1.96  S.E.) 

PPV 

NPVb 
(%) 

#  hours- 
MD  per 
QF 
detected11 

#  hours- 
nurse 
per  QP 
detected* 

UCDS 

ALL 

61.6 

85.5±6.6 

48.3+9.1 

26.5±  6.6 

93. 9±  9.2 

1.9 

6.7 

CABG 

53.9 

80.6±10.4 

51.9±13.0 

26.6  f 

92.5  f 

1.9 

7.0 

PTCA 

60.1 

88.2±9.8 

46.0±15.5 

26.1  f 

94.7  f 

1.9 

6.4 

MI 

67.7 

100.0 

39.3±11.2 

26.3  ' 

100.0  f 

1.9 

5.6 

JHH 

ALL 

55.4 

74.1  +  12.2 

56.8±11.3 

26.8±12.8 

91.1±10.9 

1.9 

1.9 

CABG 

61.6 

97.0±  3.8 

46.1  ±17.6 

28.1  f 

98.6  f 

1.8 

1.4 

PTCA 

36.8 

59.7±29.1 

68.2±17.0 

29.0  ' 

88.6  f 

1.7 

2.3 

MI 

32.3 

14.4* 

63.7+1 1.7 

8.0  f 

77.4  f 

6.3 

9.7 

HMPS 

ALL 

48.5 

69.4+18.4 

56.1±11.5 

25. 6±  12.0 

89.4±11.4 

2.0 

2.0 

CABG 

55.5 

79.1  ±29.1 

49. 6±  17.4 

25.5  f 

91.6  r 

2.0 

1.7 

PTCA 

42.6 

69.7+27.6 

63.3±18.0 

29.2  f 

90.6  f 

1.7 

2.0 

MI 

36.6 

14.4« 

58.5±12.7 

7.0  r 

75.8  f 

7.1 

9.6 

■  Positive  Predictive  Value    b  Negauve  Predictive  Value     '  Quality  Problem      d  Assuming  1/2  hour 
per  physician  review.      •  Assuming  1  hour  per  nurse  abstraction  for  UCDS,  and  15  mm.  for  JHH  and 
HMPS  (each).       r  Not  a  pooled  estimate.  Calculated  from  pooled  sensitivity,  specificity  and  overall 
prevalence.      B  No  variance  within  strata. 


Table  27 


Performance  of  different  subsets  of  UCDS  flags  in  a  set  of  medical  records  with  17.8  %  found  to  have  a  Quality  Problem.  Quality  of 
Care  Screening  Study.  Baltimore  MD,  1992/1993 


Mag 

excluded  from  set* 


%  of 
Records 
failed 
screens 


Sensitivity      Specificity        PPV 
(%)  (%)  (%) 


Complete  set' 


57.7 


85.5 


48.3 


26.5 


(%) 


93.9 


#  hours-      H  hours- 
MD  per      Nurse  per 


NPV b     QP 


1.8 


QP 


detected  c    detectedd 


6.7 


OP05B    Auemia/Polycythemia.  Acute       47.7 
care  services  not  rendered. 


60.8 


55.1 


22.4         86.5  2.2 


9.3 


CJS05II     Diagnosis  codes  include 
complications  of  procedures. 


48.5 


76.3 


57.5 


28.1  91.2  1.8 


7.3 


GS02C9    No  follow  up  of  acutely 
abnormal  X-ray. 


56.7 


81.2 


48.7 


25.7        92.3  1.9 


6.5 


OP04A     leukocytosis/leukopenia. 
Severity  of  disease  not  established. 


57.5 


85.3 


48.5 


26.5         93.9  1.9 


6.5 


OPUIA     lindocrine  (Calcium). 

Severity  of  disease  not  established. 


55.9 


85.3 


50.6 


27.4  94.1  1.8 


6.5 


continued 


Table  27 


Mag 

excluded  from  set* 


%  of 
Records 
failed 
screens 


OP07A  -  Acid  base  disturbances. 
Severity  of  disease  not  established. 


54.3 


OPIOA     Pulmonary  Disease.  Severity       57.6 
of  disease  not  established. 


Sensitivity      Specificity        PPV '      NPV b 
(*)  (%)  (%)         (%) 


84.6 


85.3 


52.3 


48.5 


#  hours- 
MD  per 

QP 

detected ' 


27.9         94.0  1.8 


26.6         93.9  1.9 


#  hours- 
Nurse  per 
QP 
detectedd 

6.6 


6.7 


OS02C24    IV  medication  discontinued      57.3 

on  day  of  discharge. 

GS06K5     Unexpected  inpatient  event         55.9 
within  3  days  of  operating  room. 

GS06K8    Cardiac  arrest  within  day  of       57.6 
blood  transfusion. 


85.3 


85.3 


85.5 


48.8 


50.5 


48.4 


26.7         93.8  1.9  6.6 


27.3         94.1  1.8  6.5 


26.5         93.8  1.9  6.6 


GS06K9     Death  within  1  day  of  blood       57.6 
transfusion. 


85.5 


48.4 


26.5         93.8  1.9 


6.6 


GS06K32     Accidental  puncture  of 
laceration  during  procedure. 


57.6 


85.5 


48.3 


26.5         93.8  1.9 


6.6 


continued 


Table  27 


Flag 

excluded  from  set* 


%of 
Records 
failed 
screens 


Sensitivity      Specificity        PPV  ■ 

(%)  (%)  (%) 


"Chance"  Perform/ 


57.7 


57.7 


42.3 


NPVb 

(%) 


tt  hours-  #  hours- 

MD  per  Nurse  per 

QP  QP 

delected  c  detected11 


17.8 


82.4 


2.8 


*  Flag  description  is  truncated.  Please,  see  Table  D-l  1  for  full  description. 

"  Positive  predictive  value.    b  Negative  predictive  value.  c  Assuming  1/2  hour  per  physician  review.  d  Assuming  I  hour  per  nurse 
abstraction.  '  The  complete  set  includes  screens  that  contributed  with  very  few  cases  and  were  not  included  in  these  analyses.   '  No 
Screening.  Proportion  taken  from  complete  set. 


Table  28 


Performance  of  subsels  of  JHH  Screens  formed  by  exclusion  of  one  screen,  in  a  sel  of  medical  records  with  17.8  %  found  to  have 
a  Quality  Problem.  Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Screen 

excluded  from  set* 


%of 

Records 

failed        Sensitivity 

screens        (%) 


Specificity      PPV" 


Complete  Set'  48.7  74.1 

1025    Complication  of  diagnostic        34.2  57.4 

or  invasive  procedure. 

154     Abnormal  diagnostic  finding        41.7  64.3 

not  addressed  and  resolved. 

800  -  Hospital-acquired  infection.         48.3 

125  -  Death  during  or  following  any     48. 1 
surgery. 

101  -  Readmission  for  complication      45.7  66.0 

or  incomplete  hospitalization. 

109  -  Care  or  lack  of  care  resulting       48.6  74. 1 

in  serious  complications. 


56.8 
70.9 

63.2 


58.7 


57.0 


NPVb 

(%) 


27.2 
30.0 

27.6 


91.0 
88.4 

89.0 


25.8 


27.3 


88.8 


91.0 


It  hours-  ft  hours- 

MD  per  Nurse  per 

QP  QP 

detected c  detected11 


1.8 

1.7 

1.8 


1.9 


1.8 


1.8 
2.4 

2.2 


74.1 

57.3 

27.5 

91.1 

1.8 

1.9 

74.4 

57.4 

27.3 

90.8 

1.8 

1.9 

2.1 


1.9 


continued 


Table  28 


Screen 

excluded  from  set* 


%of 

Records 

failed         Sensitivity      Specificity       FPV 

screens         (%)  (%)  (%) 


803  -  Hospital  acquired  decubitus         48.7 
ulcer. 

108  -  Medication  error  or  adverse         48.3 
drug  reaction. 

155  -  IV  fluids  after  12  midnight  on      47.8 
day  of  discharge. 

156  -  Purulent  or  bloody  drainage         47.3 
or  wound  or  open  area. 

301  -  Reintubalion  of  patient  within      58.7 
72  hr.  after  extubation. 

7500  -  Death  within  48  hr.  of  admit      48.4 
through  the  emergency  room. 


"Chance" 
Perform/ 


48.7 


74.1 

73.4 

70.0 

66.0 

74.1 

74.1 

48.7 


56.8 

57.1 

57.0 

56.8 

56.8 

57.2 

51.3 


27.2 

27.2 

26.2 

25.0 

27.2 

27.4 

17.9 


#  hours-  ft  hours 

Ml)  per  Nurse  per 

NPV"  QP  qp 

(%)  detected  c  delected'1 


91.0 


90.8 


89.7 


88.5 


91.0 


91.0 


82.1 


1.8 


1.8 


1.9 


2.0 


1.8 


2.8 


1.9 


1.9 


2.0 


2.1 


1.9 


1.9 


*  Description  of  screens  is  truncated.   (Camacho  1994,  Table  D-12,  contains  a  full  description  of  screens.) 


"Positive  predictive  value.    "Negative  predictive  value.    'Assuming  1/2  hour  per  physician  review.    "Assuming  1  hour  per  nurse 
abstraction.    "  Includes  flags  that  contributed  with  very  few  cases  and  are  not  listed  above.    'No  screening.    Proportion  of  failed  screens 
taken  from  complete  set. 


Table  29 


Performance  of  different  subsets  of  HMPS  Screens  in  a  set  of  medical  records  with  17  8  %  found  to  have  a  Quality  Problem. 
Quality  of  Care  Screening  Study.  Baltimore  MD,  1992/1993 


Screen 

excluded  from  set* 


%of 
Records 
failed 
screens 


Sensitivity     Specificity 


Complete  Set  48.5  69.4 

02  Subsequent  admission  related  to  index        36.6  40.7 
admission. 

03  Hospital  incurred  trauma  not  resolved,       43.0  67.5 
or  extending  hospitalization. 

04  -  Untoward  drug  reaction  unresolved  at        48.2  69.2 
discharge,  or  extending  hospitalization. 

05  -  Transfer  from  general  care  to  special         48.0  69.0 
care  unit. 

06  -  Transfer  to  acute  care  hospital.  48.5  69.4 

07  -  Unplanned  return  to  operating  room.  48.4  69.4 

08  -  Treatment  or  operation  because  of  48.7  69.4 
damage,  or  subsequent  to  invasive  proc. 

09- Acute  Ml,  CVA,  or  PE  during  or  48.4  69.4 

following  surgical  or  invasive  proc. 


PPV* 

(%) 


NPVb 
(%) 


56.1 
64.3 

62.4 

56.3 

56.5 

56.1 
56.2 
56.1 

56.1 


25.6 
19.9 

28.1 

25.6 

25.7 

25.6 
25.6 
25.6 


89.4 
83.3 

89.8 

89.4 

89.3 

89.4 
89.4 
89.4 


ft  hours-  ff  hours- 

MD  per  Nurse 

QP  per  QP 

detected c  detected11 


25.6        89.4 


2.0 
2.5 

1.8 

1.9 

1.9 

2.0 

2.0 
2.0 

2.0 


2.0 
3.4 

2.1 

2.0 

2.0 

2.0 
2.0 

2.0 

2.0 


continued 


Table  29 


Screen 

excluded  from  set 


10  -  Hospital-acquired  neurological  deficit.       48.7 

1 1  -  Death,  except  in  terminal  care,  DNR         46.6 
order,  gravely  ill  patient. 

12  Unexplained  abnormal  vital  sign  on  last     46.9 
full  day  of  discharge. 

13  Cardiac  arrest,  except  in  terminal  care,      48.3 
DNR  order,  gravely  ill  patient. 

15  -  Other  undesirable  outcome.  37.6 


"Chance" 
Perform.' 


48.5 


%of 

Records 

failed  Sensitivity     Specificity 

screens  (%)  (%) 


69.4 
69.2 

69.4 

69.4 

61.2 

48.7 


56.1 
58.3 

57.9 

56.3 

67.5 

51.3 


PPV 
(%) 


25.6 
26.5 

26.4 

25.7 

29.1 

17.9 


NPVb 
(%) 


89.4 
89.7 


82.1 


#  hours- 
MD  per 
QP 
detected 


2.0 
1.9 


89.6  1.9 

89.4  1.9 

88.9  1.7 


2.8 


ft  hours- 
Nurse 
per   QP 
detected'1 


20 
2.0 

2.0 

2.0 

2.3 


*  Screen  description  is  truncated.   (Camacho  1994,  Table  D-13,  contains  a  full  description  of  screens). 


a  Positive  predictive  value.    "Negative  predictive  value.    c  Assuming  1/2  hour  per  physician  review.    d Assuming  I  hour  per  nurse 
abstraction.    e  No  screening.    Proportion  of  failed  screens  taken  from  complete  set. 


Table  30 

Sensitivity  (%),  Specificity  (%)  and  Positive  and  Negative  Predictive  Values  (%)  pooled 
across  conditions,  by  levels  of  comorbidity  and  severity  of  illness  at  admission.    Mean  of 
estimates  from  two  reviewers.   Quality  of  Care  Screening  Study.   Baltimore  MD, 
1992/1993 


Severity 
High 

of  Illness 

Comorbidity 

UCDS 

Low 

High 

Low 

Sensitivity 

100.0 

79.1 

87.5 

83.6 

Specificity 

37.8 

58.6 

43.8 

52.4 

Pos.  Pred. 

Value' 

25.9 

29.4 

25.3 

27.7 

Neg.  Pred. 

Value  " 

100.0 

92.8 

94.1 

93.1 

JHH 

Sensitivity 

97.6 

60.3 

55.2 

96.8 

Specificity 

52.5 

62.8 

49.0 

66.6 

Pos.  Pred. 

Value1 

30.9 

26.1 

19.1 

38.7 

Neg.  Pred. 

Value* 

99.0 

87.9 

83.4 

99.0 

HMPS 

Sensitivity 

77.6 

69.9 

65.2 

82.7 

Specificity 

51.9 

61.1 

48.0 

65.2 

Pos.  Pred. 

Value* 

26.0 

28.1 

21.4 

34.1 

Neg.  Pred. 

Value* 

91.4 

90.3 

86.4 

94.5 

Calculated  based  on  a  common  prevalence  of  17.8%. 


Table  31 


Odds  ratios  (O.R.)  for  failing  UCDS  screens  for  confirmed  quality  problems:   overall  and 
stratified  estimates  by  levels  of  comorbidity  and  severity  of  illness  at  admission.   Quality 
of  Care  Screening  Study.   Baltimore  MD,  1992/1993 


Stratum 

Stratified  O.R. 

Overall  O.R.  (95%  C.I.) 

Low 

5.00 

Comorbidity  * 

High 

4.60 

Low 

4.62 

4.83  (1.18,  19.81) 

Severity  of 
Illness 

High 

_  b 

Tests  for  interaction:   '  x2,  =0.30  (p  < 0.75)  ;     b  Zero  ceil:  UCDS  =  1  predicts 
outcome  perfectly. 


Table  32 


Odds  ratios  (O.R.)  for  failing  JHH  screens  in  cases  with  a  confirmed  quality  problems: 
overall  and  stratified  by  comorbidity  and  severity  of  illness  at  admission.   Quality  of  Care 
Screening  Study.    Baltimore  MD,  1992/1993 


Stratum 

Stratified  O.R. 

Overall  O.R.  (95%  C.I.) 

Low 

39.87 

Comorbidity  * 

High 

0.95 

Low 

1.50 

2.62  (0.83.  8.29) 

Severity  of 
Illness  b 

High 

39.87 

Tests  for  interaction:   *  x2,  =21.17  (p<0.001)    b  X:,  =6.94  (p<0.01) 


Table  33 


Odds  ratios  (O.R.)  for  failing  HMPS  screens  in  cases  with  a  confirmed  quality  problems: 
overall  and  stratified  by  comorbidity  and  severity  of  illness  at  admission.   Quality  of  Care 
Screening  Study.    Baltimore  MD,  1992/1993 


Stratum 

Stratified  O.R. 

Overall  O.R.  (95%  C.I.) 

Low 

6.07 

Comorbidity  * 

High 

1.44 

Low 

2.17 

2.54(0.81,  7.96) 

Seventy  of 

High 

3.54 

Illness  b 

Tests  for  interaction:  '  x\  =1.35  (p<0.25)    b  x\  =  1.80  (p<0.25) 


Table  34 


Estimated  odds  ratios  (O.R.)   and  95%  confidence  intervals  tor  individual  UCDS 
screens  which  best  predicted  quality  problems.  Quality  of  Care  Screening  Study. 
Baltimore  MD,  1992/1993 


Variable 

Univariate  O.R. 

(95%  C.I.) 

Adjusted  O.  R. 
(95%  C.I.) 

Seventy  of  Illness 

at  Admission 

0.55 

(0.19- 

1.58) 

0.28   (0.09-    0.92) 

Comorbidity 

1.24 

(0.44  - 

3.51) 

1.45   (0.46-    4.57) 

GS6K32 

20.33 

(2.48  - 

143.52) 

28.12   (4.51  -175.12) 

GS2C24 

5.64 

(0.83  - 

38.28) 

15.14   (1.70-134.72) 

GS6K31 

1.72 

(0.38  - 

7.84) 

10.72   (1.62  -  70.96) 

GS2C9 

4.13 

(0.71  - 

24.12) 

10.72   (1.47-78.43) 

GS6K8 

1.18 

(0.29  - 

4.79) 

8.46  (1.53  -  46.88) 

OP05B 

2.52 

(0.79  - 

8.11) 

3.71    (1.01  -    13.65) 

GS6K5 

0.21 

(0.06  - 

0.73) 

0.15   (0.03-    0.79) 

GS6K10 

0.19 

(0.02  - 

1.54) 

0.06  (0.00-    0.96) 

Table  35 


Estimated  odds  ratios  and  95%  confidence  intervals  for  individual  JHH  screens  which 
best  predicted  quality  problems.  Quality  of  Care  Screening  Study.  Baltimore   MD, 
1992/1993 


Variable 

Univariate  O.R 
(95%  C.I.) 

Adjusted  O.R. 
(95%  C.I.) 

Severity  of  Illness 

at  Admission 

0.55  (0.19- 

1.58) 

0.07    (0.03  - 

0.18) 

Comorbidity 

1.24  (0.44- 

3.51) 

1.71    (0.43- 

6.77) 

127 

1.66  (0.47- 

5.85) 

14.68    (3.15- 

68.32) 

155 

17.59  (1.96- 

157.78) 

232.47     (21.53  - 

2509.61) 

156 

28.67  (3.39- 

242.14) 

34.35    (4.56- 

258.48) 

151 

5.45   (0.39- 

75.92) 

28.77    (4.23  - 

195.46) 

109 

6.57  (1.01  - 

42.75) 

11.64    (2.17- 

62.28) 

108 

1.04  (0.40- 

2.68) 

3.32    (1.04- 

10.63) 

Table  36 


Estimated  odds  ratios  and  95%  confidence  intervals  for  individual  HMPS  screens  which 
best  predicted  quality  problems.  Quality  oi  Care  Screening  Study.  Baltimore   MD, 


1992/1993 

Variable 

Univariate  O.R. 

Adjusted  O.R. 

(95%  C.I.) 

(95%  C.I.) 

Severity  of  Illness 

at  Admission 

0.55   (0.19-    1.58) 

0.38 

(0.11  -  1.34) 

Comorbidity 

1.24   (0.44-   3.51) 

1.28 

(0.40-  4.11) 

02 

3.98  (1.16-  13.62) 

4.38 

(1.06-  18.16) 

05 

5.23   (1.35  -20.30) 

8.02 

(1.53-42.01) 

07 

0.12  (0.02-  0.92) 

0.13 

(0.02-    1.00) 

10 

0.16  (0.02-    1.28) 

0.10 

(0.01  -    1.09) 

13 

0.29   (0.10-  0.88) 

0.29 

(0.06-    1.35) 

Table  Appendix  page 

A1 .    Listing  of  adverse  events  and  quality  1 

problems  by  record  ID,  Ml  patients  (N=90) 


A2.    Listing  of  adverse  events  and  quality 
problems  by  record  ID,  PTCA  patients 
(N  =  134) 


A3.    Listing  of  adverse  events  and  quality  14 

problems  by  record  ID,  CABG  patients 
(N=213) 


Abbreviations  used  in  Appendix  tables  26 


Quality  of  Care  Screening  Study 
A1.       Listing  of  adverse  events  and  quality  problems  by  record  ID,  Ml  patients  (N=90) 


QID 


001 


002 


003 


004 


005 


006 


007 


008 


009 


010 


011 


012 


013 


014 


015 


Adverse  Event 


Not  specified 


None 


Not  specified 


Sepsis;  Gl  bleed 


None 


Recurrent  ischemia 


Cardiac  arrest;  Gl  bleed 


None 


Gl  bleed 


2nd  Ml  after  Gl  bleed;  renal  failure 


None 


Postoperative  bleeding  requiring 
re-operation 


Bleeding  from  cath  site 


Worsening  renal  failure 


EMD  arrest 


None 


Quality  Problem 


None 


Abnormal  lab  test  not  addressed 


None 


None 


None 


Heparin  not  given 


Failure  to  install  pacemaker  or  to 
perform  ECG  monitoring;  failure  to 
perform  endoscopy 


On  antimicrobial,  no  culture 
ordered 


Unchecked  hematocrit  fall  (on 
heparin) 


Significant  hematocrit  drop 
discovered  on  the  next  day 


None 


Not  described 


Abnormal  lab  value  not  addressed; 
premature  removal  of  sheath 


Delay  in  sending  patient  to  ICU; 
premature  discharge  while 
creatinine  was  increasing 


Failure  to  perform  cath 
None 


Quality  of  Care  Screening  Study 


QID* 

Adverse  Event 

Quality  Problem 

016 

Groin  pseudoaneurysm 

None 

017 

None 

None 

018 

Bacteremia  following  cath 

Not  specified 

019 

None 

None 

020 

None 

None 

021 

None 

None 

022 

None 

None 

Transient  visual  field  defect 

Heparin  discontinued  prematurely 

023 

Recurrent  VT  arrest  while  being 
loaded  with  amiodarone 

Premature  discharge  from  CCU 
with  return  24  hours  later 

024 

Reocclusion  of  LAD 

Inadequate  anticoagulation 

025 

None 

None 

026 

None 

None 

027 

None 

None 

028 

AMI  during  PTCA  attempt 

None 

029 

None 

None 

030 

None 

None 

031 

None 

None 

032 

Ml;  aspiration  pneumonia  and 
sepsis 

None 

033 

None 

None 

101 

None 

None 

102 

Acute  Gl  bleeding;  Nosocomial 
infection  (C.  difficile  colitis) 

None 

Quality  of  Care  Screening  Study 


QID* 

Adverse  Event 

Quality  Problem 

103 

None 

None 

104 

None 

None 

106 

Nosocomial 

infection 

(UTI) 

None 

107 

None 

None 

108 

None 

None 

109 

None 

None 

110 

None 

None 

111 

None 

Prolonged  and  inappropriate  delay 
in  consult  and  transfer  to  CCU 

112 

Acute  renal  failure 

Delay  in  transfer  to  CCU 

113 

None 

None 

114 

None 

No  risk  stratification  post-MI  with 
known  70-90%  LAD  stenosis 

115 

None 

None 

116 

None 

None 

117 

None 

None 

118 

None 

None 

119 

None 

None 

120 

Stroke 

Failure  to  heparinize  fresh  Ml 

121 

None 

None 

122 

None 

None 

123 

None 

None 

124 

Gross  hematuria  from  thrombolytic 

None 

Quality  of  Care  Screening  Study 


QID 


125 


126 


127 


128 


129 


130 


131 


132 


133 


135 


136 


137 


138 


139 


140 


141 


142 


143 


144 


Adverse  Event 


None 


None 


None 


None 


Line  sepsis 


None 


None 


None 


None 


Heart  block  necessitating 
pacemaker 


None 


None 


None 


None 


Post  CABG  A-fib 


None 


None 


None 


None 


None 


Quality  Problem 


None 


None 


None 


None 


None 


None 


None 


None 


None 


Poor  initial  medical  treatment 


None 


None 


Abnormal  diagnostic  finding  not 
addressed 


None 


Drug  given  and  blood  level  not 
measured;  symptomatic 
bradycardia  and  no  Holter  or 
pacemaker 

None 

None 

None 

None 

None 


Quality  of  Care  Screening  Study 


QID* 

Adverse  Event 

Quality  Problem 

145 

None 

None 

146 

Hypoglycemia  X2 

Insulin  dose  not  adjusted  at 
discharge 

147 

None 

Delay  in  tPA  therapy 

148 

None 

None 

149 

None 

None 

150 

None 

None 

151 

None 

None 

152 

None 

None 

153 

None 

None 

154 

None 

None 

155 

None 

None 

156 

None 

None 

157 

None 

None 

158 

None 

None 

159 

None 

Discharge  without  stress  test 

The  QID  is  the  record  identification  number.   Records  #001-033  had  failed 
screening  indicators  at  JHH;  records  #101-159  had  passed  it.   Numbering  is 
discontinuous  as  the  leftmost  digit  identifies  the  study  stratum.   Some  strata  had 
less  than  99  cases.  The  two  rightmost  digits  of  the  QID  range  between  01  and 
99;  00  is  not  used.   Forms  from  record  QID  #105  were  unavailable  for  review. 


Quality  of  Care  Screening  Study 

A2.       Listing  of  adverse  events  and  quality  problems  by  record  ID,  PTCA  patients 
(N  =  134) 


QID* 

Adverse  Event 

Quality  Problem 

201 

Retroperitoneal  hemorrhage  after 
groin  procedure 

None 

202 

Fall  in  hematocrit  2nd  to  groin 
hematoma 

None 

203 

Gl  bleeding 

No  Gl  prophylaxis  despite  stress  in 
CCU,  thrombolytics,  ASA,  heparin 

204 

Groin  hematoma 

None 

205 

VT  during  RCA  injection 

Over-sedation  with  Valium, 
respiratory  depression;  no  ABG 

206 

Small  Ml 

Unclear 

208 

None 

None 

209 

Coronary  occlusion  after  PTCA; 
femoral  bleeding  after  PTCA 

None 

210 

Groin  hematoma;  Gl  bleeding 

None 

211 

CHF 

Fluid  overload  from  transfusion 

212 

Groin  bleeding,  hematoma, 
requiring  blood  transfusion, 
following  thrombolytic  therapy  and 
cath 

None 

213 

Adverse  drug  reaction; 
streptokinase 

None 

214 

Small  Ml  during  PTCA 

Abnormal  diagnostic  finding  not 
addressed 

215 

Pulmonary  edema 

Poor  fluid  management 

Quality  of  Care  Screening  Study 


QID 


216 


217 


219 


224 


225 


226 


230 


Adverse  Event 


Retroperitoneal  bleed 


Retroperitoneal  Bleed 


Coronary  occlusion  and  thrombus 
during  PTCA 


VT/VF  arrest  during  cath 


Ventricular  arrhythmias,  following 
acute  Ml  during  PTCA  attempt 

Acute  closure  during  PTCA; 
hematoma  with  falling  hematocrit 


Ischemia  lower  extremities  following 
cath,  IABP,  PTCA 


Groin  bleed  with  hematoma 
requiring  transfusion,  and  Ml  during 
PTCA 


None 


DVT  following  cath/PTCA 


Coronary  reocclusion  s/p  PTCA 


Retroperitoneal  bleeding;  fall  in 
hematocrit 


Retroperitoneal  bleed;  CHF  post 
transfusion 


Quality  Problem 


Premature  discharge;  CT  to  localize 
pseudoaneurysm  not  done; 
hematocrit  not  checked  on 
discharge 


Heparin/PTT  not  well  controlled 


Unclear 


None 


None 


Abnormal  lab  finding  not  addressed 
(discharged  with  hematocrit  still 
falling) 


None 


Poor  evaluation  and  follow-up  of 
hematoma  before  discharge 


None 


Thrombolytic  therapy  not  clearly 
indicated 


PTCA  not  indicated 


Inadequate  evaluation  of  rapid 
hematocrit  fall 


Large  hematoma  with  low 
hematocrit 


Groin  hematoma;  small  Ml 


Inadequate  management  of  fluid 
overload 


Unclear 


None 


Quality  of  Care  Screening  Study 


QID  * 

Adverse  Event 

Quality  Problem 

231 

Groin  hematoma;  femoral  nerve 
palsy 

None 

232 

Renal  failure  post  cath 

Poor  fluid  management 

233 

Groin  pseudoaneurysm 

None 

234 

Bleeding  following  thrombolytic 
therapy 

None 

235 

Bleeding  post  cath 

None 

236 

Gl  bleeding;  allergic  reaction  to 
thrombolytic 

None 

237 

Pulmonary  edema  due  to  LAD 
closure;  acute  renal  failure  due  to 
dye 

None 

238 

Groin  bleed  and  fall  in  hematocrit 

None 

239 

Cerebrovascular  hemorrhage 
following  thrombolytic 

None 

240 

Groin  bleeding;  fat  necrosis 

None 

241 

Groin  hematoma  leading  to  anemia; 
pseudoaneurysm 

None 

242 

Fall  in  hematocrit  and  hematoma; 
RCA  dissection  during  PTCA 

None 

243 

Coronary  dissection  during  PTCA; 
groin  infection;  line  infection 

None 

244 

None 

None 

245 

VF  arrest  during  cath;  fall  in 
hematocrit  during  cath. 

None 

246 

Subarachnoid  bleed 

tPA  given  after  major  head  trauma 
and  epistaxis 
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Quality  of  Care  Screening  Study 


QID  * 

Adverse  Event 

Quality  Problem 

247 

Staphylococcus  aureus  bacteremia 
with  infected  hematoma 

None 

248 

VF  arrest;  sternal  wound 
dehiscence 

None 

249 

Bleeding  at  cath  site  requiring 
transfusion 

Unclear 

250 

Acute  renal  failure  secondary  to 
contrast;  thrombophlebitis 
secondary  to  IV  catheter 

None 

251 

VF  arrest  during  cath 

None 

252 

Groin  bleeding  and  surgery  at  IABP 
site 

None 

253 

Dissection  and  Ml  during  PTCA 

None 

254 

Groin  pseudoaneurysm 

None 

255 

Exacerbation  of  chronic  renal 
failure;  Ml  after  cath 

None 

257 

AV  fistula  after  cath 

None 

258 

Acute  renal  failure,  CHF  due  to  IV 
contrast  and  volume 

Gentamicin  given  prior  to  cath; 
misuse  of  contrast 

259 

Total  occlusion  of  RCA  following 
failed  PTCA 

Consideration  of  CABG  was  not 
documented 

260 

CHF,  UTI,  groin  bleed 

Poor  fluid  management 

261 

VF  arrest  during  cath;  groin  bleed 
requiring  transfusion 

None 

262 

Pulmonary  edema  during  cath 

None 

263 

Line  sepsis;  deterioration  of  renal 
function 

Renal  function  deterioration  not 
addressed 

Quality  of  Care  Screening  Study 


QID 


264 


266 


267 


269 


270 


271 


272 


273 


274 


275 


276 


277 


278 


280 


281 


282 


Adverse  Event 


Protamine  reaction 


Right  femoral  artery 
pseudoaneurysm 


Cardiac  arrest  during  PTCA 


Groin  pseudoaneurysm 


RCA  dissection  and  Ml  post 
angioplasty 


Acute  renal  failure 


None 


CVA  after  cath 


Large  groin  hematoma 


Groin  hematoma 


RCA  restenosis 


Acute  dissection  and  Ml 


Pseudoaneurysm 


Acute  Ml  due  to  acute  occlusion 


Line  sepsis 


Laceration  of  right  femoral  artery; 
pulmonary  edema 


None 


Bradycardia  and  hypotension 


Bleeding  at  cath  site 


Retroperitoneal  bleed;  Ml  s/p 
angioplasty 


Quality  Problem 


None 


Abnormal  test  result  not  addressed 


Trauma  from  guide  wire  in  LV 


None 


None 


None 


None 


None 


None 


None 


None 


None 


None 


Anemia  not  addressed 


None 


None 


None 


Untreated  UTI 


Puncture  of  femoral  profunda  artery 


None 
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QID  * 

Adverse  Event 

Quality  Problem 

284 

Line  sepsis 

Probable  break  in  sterile  technique 

286 

Bacteremia 

None 

287 

Pseudoaneurysm 

None 

288 

Ml  post  PTCA 

None 

289 

None 

None 

290 

Groin  hematoma 

None 

291 

Ml,  2nd  left  circumflex  artery 
dissection  during  PTCA 

Discharge  planning:  antibiotics 
decided  but  not  prescribed 

292 

Pulmonary  edema  and  increased 
CPK  during  cath 

None 

293 

Line  sepsis 

Discharge  with  elevated  WBC  and 
fever 

294 

Femoral  artery  dissection;  aspiration 

Over-sedation;  failure  to  monitor 
patient  hemodynamically  in  ICU 

295 

None 

None 

296 

Intracranial  hemorrhage  due  to 
streptokinase 

None 

297 

Occlusion  of  subtotal  circumflex 
lesion  during  cath 

Periocardiocentesis  not  attempted 

298 

Sudden  asystolic  death 

Undetermined 

299 

Left  arm  hematoma  requiring 
transfusion,  left  circumflex  artery 
dissection  requiring  CABG  [X4] 

None 

301 

None 

CABG  not  considered  where 
lesions  indicated 

302 

None 

None 
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QID  * 

Adverse  Event 

Quality  Problem 

303 

None 

None 

304 

Small  hematoma  and  mild 
hematocrit  fall 

None 

305 

None 

None 

309 

Anemia  requiring  transfusion 

Inadequate  anticoagulation 
post-tPA 

310 

None 

None 

311 

Fall  in  hematocrit  after  PTCA 

None 

312 

Reocclusion  of  LAD  after  PTCA 

Discharged  with  low  blood 
pressure  without  physical 
examination 

313 

None 

None 

314 

None 

None 

315 

Recurrent  chest  pain  after  PTCA 

None 

316 

None 

None 

318 

None 

None 

319 

None 

None 

320 

None 

None 

321 

CHF;  anemia 

Volume  overload 

322 

None 

None 

323 

None 

None 

325 

None 

None 

326 

Anemia 

Discharged  without  addressing 
anemia 

328 

None 

None 
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QID* 

Adverse  Event 

Quality  Problem 

329 

None 

None 

330 

None 

None 

331 

None 

None 

332 

None 

None 

901 

Retroperitoneal  hematoma  s/p 
PTCA 

None 

902 

Pericardial  tamponade 

Perforation  of  left  circumflex  artery 
during  PTCA 

903 

RCA  dissection  during  PTCA 

None 

904 

Renal  failure;  LIMA  dissection 

None 

905 

Acute  LAD  reocclusion  following 
cessation  of  heparin 

None 

906 

Reocclusion  and  small  Ml  post 
PTCA 

None 

907 

Groin  hematoma  and  drop  in 
hematocrit 

Hematocrit  not  addressed 

908 

Groin  hematoma  following  PTCA 

None 

909 

None 

None 

910 

Pneumonia 

None 

911 

Cardiac  arrest  during  cath 

None 

912 

None 

None 

The  QID  is  the  record  identification  number.    Records  #201-299  and  #901-912 
failed  screening  at  JHH;  records  #301-332  had  passed  it.   Numbering  is 
discontinuous  as  the  leftmost  digit  identifies  the  study  stratum.   Some  strata  had 
less  than  99  cases.   The  two  rightmost  digits  of  the  QID  range  between  01  and 
99;  00  is  not  used.    Review  forms  for  records  QID  #207,  256,  285,  306,  307, 
308,  317,  324,  and  327  were  unavailable  for  review,  while  three  additional 
medical  records  were  unavailable  to  study  reviewers. 
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Appendix  A3.  Listing,  of  adverse  events  and 
patients  (N  =  213) 


Quality  of  Care  Screening  Study 
quality  problems  by  record  ID,  CABG 


QID* 

Adverse  Event 

Quality  Problem 

401 

Small  Ml  during  CABG 

None 

402 

Ruptured  diverticulum 

None 

403 

Perioperative  CVA 

Poor  antibiotic  choice;  poor 
respiratory  management 

404 

Postoperative  bleeding 

Poor  use  of  antibiotics 

405 

Sternal  and  leg  harvest  site 
infection;  VF  arrest 

None 

406 

Cellulitis 

Cellulitis  postoperative 

407 

Arterial  puncture  during  CABG 

No  daily  progress  note  in  chart 

408 

None 

None 

409 

Neurologic  changes 

None 

410 

Stroke 

Lack  of  progress  notes  2  days 
prior  to  discharge;  worsening  of 
creatinine  not  resolved  by 
discharge;  heparin  dose  error 

411 

Pneumonia 

Cardiac  medication  without 
indication 

412 

None 

None 

413 

None 

None 

414 

Postoperative  death 

None 

415 

Sternal  wound  infection 

Poor  management  of  fluid  status 

416 

A-fib 

None 

417 

None 

None 

418 

None 

None 
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QID* 

Adverse  Event 

Quality  Problem 

419 

Sacral  decubitus  ulcer 

Diabetic,  glucose  <85  within  2 
days  of  discharge 

420 

Pneumothorax 

None 

421 

Cardiac  arrest 

Positive  culture  of  central  line 

422 

Death  following  Ml  and  CABG 

None 

423 

Perforated  diverticulum 

None 

424 

Sepsis 

Delay  in  medical  consultation 

425 

Mediastinitis 

Drop  in  hematocrit  not  addressed 

426 

None 

None 

427 

DVT 

No  postoperative  subcutaneous 
heparin 

428 

Perioperative  Ml 

Discharged  without  beta  blocker; 
no  post  Ml  risk  assessment 

429 

Postoperative  leg  infection 

Poor  management  of  infection 

430 

Stroke 

None 

431 

Femoral  vein  laceration 

None 

432 

Acute  renal  failure 

None 

433 

Stroke;  aspiration;  ARDS 

No  physician  note  documenting 
adverse  clinical  event 

434 

Pneumothorax 

None 

435 

Hypoxemia  requiring  intubation; 
resp.  arrest 

Pulmonary  embolism  not 
considered 

436 

Death 

None 

437 

Postoperative  mediastinal  bleeding 
requiring  re-operation 

Creatinine  significantly  higher  at 
discharge 
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QID* 

Adverse  Event 

Quality  Problem 

438 

Postoperative  mediastinitis 

None 

439 

Intraoperative  stroke;  wound 
dehiscence 

None 

440 

Stroke  after  CABG 

None 

441 

Multiorgan  system  failure 

None 

442 

Postoperative  stroke 

None 

443 

Sternal  dehiscence 

None 

444 

Laceration  of  intercostal  artery; 
multiorgan  system  failure 

None 

445 

Postoperative  respiratory  failure 

Failure  to  broaden  antibiotic 
coverage  in  a  timely  fashion 

446 

VT/VF  arrest 

None 

447 

Cardiogenic  shock 

Delay  in  IABP  insertion 

448 

None 

None 

449 

Stroke 

No  plan  for  respiratory  treatment 
after  discharge;  large  bolus  of 
heparin 

450 

None 

None 

451 

Recurrent  ischemia  following  acute 
closure  of  bypass  graft 

None 

452 

DVT 

None 

453 

Laceration  of  old  bypass  graft 

None 

454 

Postoperative  stroke 

Poor  documentation  of  home  care 
referral 

455 

Death 

None 

456 

Postoperative  vascular  insufficiency 

None 
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QID 


457 


459 


460 


461 


462 


464 


465 


466 


468 


469 


470 


472 


474 


475 


Adverse  Event 


None 


Postoperative  pancreatitis;  necrotic 
IV  site 


Graft  reocclusion 


Postoperative  VF  arrest 


Pulmonary  embolus 


GN  line  sepsis 


Arterial  occlusion  (needed 
embolectomy);  Ml  after  cath 


Cellulitis 


Renal  failure;  pneumonia 


None 


Bleeding  during  CABG  requiring 
reexploration 


Postoperative  acute  renal  failure 


Intraoperative  Ml 


Postoperative  VT  and  death 


Postoperative  bleeding  needing 
reexploration 


CHF;  ischemia  of  gut  and  liver 


Postoperative  Ml;  vocal  cord 
paralysis 


Large  stroke;  aspiration  pneumonia 


Postoperative  arrhythmia 


Quality  Problem 


None 


IV  site  not  checked 


Thyroid  supplement  without  T4  or 
TSH 


None 


Postoperative  subcutaneous 
heparin 


Break  in  sterile  technique 


None 


None 


None 


None 


None 


None 


None 


None 


None 


None 


None 


None 
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QID* 

Adverse  Event 

Quality  Problem 

476 

Postoperative  bilateral  femoral 
embolus 

None 

477 

Sternal  wound  infection 
readmission 

needing 

No  physician  response  to  nurse 
note  of  erythema  in  sternum 

478 

Intraoperative  stroke 

None 

479 

A-fib  with  rapid  ventricular  response 

None 

480 

Postoperative  bleeding  requiring 
reexploration 

None 

481 

Brachial  plexus  injury 

None 

482 

Intraoperative  stroke 

None 

483 

Stroke  after  CABG 

Carotid  screening  test  interpreted 
by  non-MD 

484 

Intraoperative  stroke 

Abrupt  withdrawal  of 
benzodiazepene  in  Ativan- 
dependent  patient 

485 

None 

None 

486 

Fractured  tooth  due  to 

ntubation 

None 

487 

Postoperative  VT/VF 

488 

None 

None 

489 

None 

None 

490 

Thrombosed  vein  graft 

Failure  to  assess  graft  before 
closing 

491 

stroke 

None 

492 

Cellulitis  in  vein  harvest  site 

None 

493 

None 

None 

494 

None 

None 
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QID  * 


495 


496 


497 


498 


499 


501 


502 


503 


504 


505 


506 


508 


509 


510 


511 


512 


513 


514 


Adverse  Event 


Cardiac  arrest 


None 


Postoperative  stroke 


None 


Postoperative  Ml 


None 


None 


None 


A-fib 


None 


A-fib 


Worsening  renal  failure  requiring 
dialysis 


None 


None 


Lower  extremity  hematoma 


Jaw  injury  from  intubation 


None 


Postoperative  bleeding 


None 


Intra-cardiac  foreign  body;  recurrent 
VT 


Quality  Problem 


Toxic  blood  levels  of  procainamide 


None 


Decubitus  ulcer 


None 


CPK  with  MB:  no  action  taken 


None 


None 


None 


None 


No  medical  notes  for  2  days  before 
discharge 


Inadequate  follow-up 


None 


None 


None 


Oozing  before  discharge  not 
addressed 


Low  hematocrit  not  addressed 


None 


No  CXR;  readmitted  with  pleural 
effusion 


None 


Intracardiac  foreign  body 
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QID  * 

Adverse  Event 

Quality  Problem 

516 

Hyperkalemia  and  sudden  death 

Inadequate  monitoring 

517 

None 

None 

518 

None 

None 

519 

None 

None 

520 

None 

None 

521 

Postoperative  bleeding 

None 

522 

Fever  presumably  2nd  to  atelectasis 

Preventable  with  pulmonary  toilet 

523 

Altered  mental  status 
postoperatively 

None 

524 

None 

None 

525 

None 

Reason  for  no  transfusion  not 
documented 

526 

Postoperative  cholecystitis 

None 

527 

None 

None 

528 

None 

None 

529 

Draining  wound  at  harvest  site 

Delay  in  addressing  cellulitis 

530 

Anemia  post  CABG 

Decreasing  hematocrit  during 
hospitalization 

531 

None 

Abnormal  p02  not  followed  up 

532 

None 

None 

533 

None 

None 

534 

Intra/postoperative  Ml 

Inappropriate  insulin  therapy 

535 

None 

None 
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QID* 

Adverse  Event 

Quality  Problem 

601 

Ml  after  CABG 

Possible  technical  problem  in 
surgery 

801 

Postoperative  tamponade  and 
stroke 

None 

802 

Pulmonary  embolism 

None 

803 

Aspiration  after  extubation 

Inappropriate  CPT:  patient  refusal; 
patient  drowsy  (Compazine); 
patient  in  Trendelenburg  position 

804 

Airway  trauma 

Multiple  unsuccessful  attempts  to 
intubate 

804 

None 

Supplemental  insulin  until  discharge 
but  not  discharged  on  insulin 

805 

Bleeding  during/following  CABG 

None 

806 

Progressive  hypotension  and 
cardiac  arrest,  required  re-do  of 
bypass,  IABP 

807 

RCA  dissection  ???  PTCA 

None 

808 

Transfusion-induced  hemolytic 
anemia 

None 

809 

Arterial  bleeding  from  graft 

Failure  to  recognize  bleeding 
promptly 

810 

Urinary  retention 

None 

811 

VT  arrest  during  CABG 

None 

812 

Pleural  effusion  following 
CABG/aortic  valve  replacement 

None 

813 

VF  and  shock  s/p  CABG 

Increased  lidocaine  to  toxic  levels 
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QID 


813 


814 


815 


818 


821 


822 


823 


825 


826 


828 


829 


Adverse  Event 


Pleural  effusion  following 
CABG/aortic  valve  replacement 


VF  and  shock  s/p  CABG 


Ml/arrest  following  CABG 


Sternal  infection  post-CABG 


Hematoma  and  fall  in  hematocrit 
post-CABG 


Arterial  thrombus  on  cath  site; 
aspiration 


Allergy  to  streptokinase 


Post-angiogram  hypotension:  CCU 
transfer 


Altered  mental  status  following 
CABG 


None 


None 


Intraoperative  stroke 


Sternal  wound  infection  during 
CABG 


None 


Sepsis  and  wound  dehiscence 


VF  arrest  and  complete  heart  block 
post-CABG 


Intraoperative  Ml  following  VF 


Arrest  after  bypass 


Quality  Problem 


None 


Increased  lidocaine  to  toxic  levels 


Inadequate  sedation;  inadequate 
exploration  of  CP 


None 


None 


Delay  in  detection  of  NG  tube 
misplacement 


None 


None 


None 


None 


None 


Unclear 


None 


None 


None 


None 


None 


None 
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QID* 

Adverse  Event 

Quality  Problem 

830 

Postoperative  respiratory  distress; 
transient  renal  insufficiency  following 
bypass 

Lack  of  follow-up 

831 

Postoperative  stroke;  allergy  to 
penicillin 

Allergy  to  penicillin  reported,  given 
ampicillin 

832 

Wound  dehiscence  after  CABG; 
postoperative  arrhythmias 

None 

833 

Respiratory  arrest;  renal  failure; 
pericarditis 

None 

834 

Postoperative  urinary  retention; 
postoperative  bleeding 

Unsecured  branch  of  LIMA 
requiring  emergent  re-operation 

835 

Postoperative  bleeding  causing 
tamponade 

Problem  in  surgical  technique 

836 

Postoperative  stroke  and  respiratory 
distress 

None 

837 

Stroke  during  CABG 

None 

838 

Postoperative  heart  block 

Incorrect  (not  ordered)  medication 

839 

Bilateral  pneumothorax  after 
removal  of  mediastinal  tube 

Removal  (premature?)  of  chest 
tube  requiring  reinsertion 

840 

Sternal  wound  dehiscence; 
postoperative  respiratory  failure 

Lack  of  documentation  of  several 
ICU  days 

841 

Complete  heart  block  during 
cardiac  cath 

None 

842 

None 

None 

843 

Postoperative  pneumonia/sepsis 

None 

844 

Postoperative  graft  thrombosis  and 
Ml 

None 

845 

Postoperative  stroke;  aortic  tear 

None 
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QID* 

Adverse  Event 

Quality  Problem 

846 

Post/intraoperative  stroke 

None 

847 

Shearing  of  catheter  tip 

Shearing  of  catheter  tip 

848 

Aortic  rupture 

None 

849 

Postoperative  foot  drop 

None 

850 

Vein  harvest  site  dehiscence 

Surgical  drain  sutured  to  wound 
closure 

851 

None 

None 

852 

Foot  drop;  altered  mental  status 

No  physician  notes  last  3  days 

853 

Hand  weakness  and  paresthesia 

None 

854 

None 

No  physician  notes  prior  to 
discharge 

855 

Injury  during  intubation 

None 

856 

AV  fistula  from  cath 

None 

857 

Pulmonary  edema  post-transfusion 

None 

858 

None 

None 

859 

Mild  hand  weakness  from  TIM 

Incorrect  hepatization 

860 

Atrial  and  ventricular  arrhythmia; 
arrest 

Failure  to  monitor  patient 

861 

VT 

Failure  to  intubate  electively 

862 

None 

None 

863 

Renal  failure 

None 

864 

Sternal  wound  infection;  pulmonary 
failure 

Inadequate  follow-up 

865 

Disseminated  intravascular 
coagulation 

None 
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QID* 

Adverse  Event 

Quality  Problem 

868 

Scalp  abrasion/cellulitis 

None 

869 

Hematuria  after  Foley  catheter 
insertion 

None 

870 

None 

None 

871 

Sepsis 

None 

872 

None 

None 

873 

None 

None 

874 

Rotator  cuff  tear  following  CT 

Delay  in  diagnosis  of  cuff  tear 

875 

Stroke  during  CABG 

Carotid  occlusion  not  explored 
adequately 

876 

Intraoperative  stroke 

None 

877 

None 

Elevation  of  creatinine/BUN  not 
discussed  in  notes 

The  QID  is  the  record  identification  number.   Records  #401-499,  #601,  and  80I- 
877  had  failed  screening  indicators  at  JHH;  records  #501-537  had  passed  it. 
Numbering  is  discontinuous  as  the  leftmost  digit  identifies  the  study  stratum. 
Some  strata  had  less  than  99  cases.  The  two  rightmost  digits  of  the  QID  range 
between  01  and  99;  00  is  not  used. 
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Abbreviations  used  in  Appendix  tables 

A-fib  atrial  fibrillation 

ABG  arterial  blood  gas  measurement 

AMI  acute  myocardial  infarction 

ARDS  acute  respiratory  distress  syndrome 

AV  aspirin 

cath  arteriovenous 

CCU  cardiac  catheterization 

CHF  coronary  care  unit 

CPK  creatine  phosphokinase  enzyme  level 

CPT  chest  physiotherapy 

CT  computed  tomography 

CXR  chest  x-ray 

DVT  deep  venous  thrombosis 

ECG  electrocardiogram 

EMD  electromechanical  dissociation 

Gl  gastrointestinal 

IABP  intraaortic  balloon  pump 

ICU  intensive  care  unit 

IV  intravenous 

lab  laboratory 

LCA  left  coronary  artery 

LIMA  left  internal  mammary  artery 

MB  myocardial  band  (of  CPK  enzyme) 

Ml  myocardial  infarction 

NG  nasogastric 

p02  partial  pressure  of  oxygen 

PTT  partial  thromboplastin  time 

RCA  right  coronary  artery 

s/p  status  post 

T4  thyroxin 

tPA  tissue  plasminogen  activator  (thrombolytic) 

TSH  thyroid  stimulating  hormone 

UTI  urinary  tract  infection 

VF  ventricular  fibrillation 

VT  ventricular  tachycardia 

WBC  white  blood  cell  count 
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